[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250831 13:56:54.124634 19366 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.233.190:40083
I20250831 13:56:54.126096 19366 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250831 13:56:54.127133 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:56:54.143519 19373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:56:54.143572 19372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:56:54.143586 19375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:56:55.302461 19374 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250831 13:56:55.302588 19366 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250831 13:56:55.303884 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:56:55.304106 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:56:55.304251 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648615304235 us; error 0 us; skew 500 ppm
I20250831 13:56:55.305042 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:56:55.309979 19366 webserver.cc:480] Webserver started at http://127.18.233.190:44143/ using document root <none> and password file <none>
I20250831 13:56:55.310832 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:56:55.311018 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:56:55.311399 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:56:55.317087 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/master-0-root/instance:
uuid: "525c9ed220ce4bb79c08c08ca2199b07"
format_stamp: "Formatted at 2025-08-31 13:56:55 on dist-test-slave-sg9n"
I20250831 13:56:55.324285 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250831 13:56:55.329139 19381 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:55.330157 19366 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250831 13:56:55.330432 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/master-0-root
uuid: "525c9ed220ce4bb79c08c08ca2199b07"
format_stamp: "Formatted at 2025-08-31 13:56:55 on dist-test-slave-sg9n"
I20250831 13:56:55.330760 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:56:55.378224 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:56:55.379655 19366 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250831 13:56:55.380105 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:56:55.448745 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.190:40083
I20250831 13:56:55.448845 19432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.190:40083 every 8 connection(s)
I20250831 13:56:55.455446 19433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:55.480835 19433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07: Bootstrap starting.
I20250831 13:56:55.486536 19433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:55.488329 19433 log.cc:826] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07: Log is configured to *not* fsync() on all Append() calls
I20250831 13:56:55.492656 19433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07: No bootstrap required, opened a new log
I20250831 13:56:55.512064 19433 raft_consensus.cc:357] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525c9ed220ce4bb79c08c08ca2199b07" member_type: VOTER }
I20250831 13:56:55.512593 19433 raft_consensus.cc:383] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:55.512835 19433 raft_consensus.cc:738] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 525c9ed220ce4bb79c08c08ca2199b07, State: Initialized, Role: FOLLOWER
I20250831 13:56:55.513628 19433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525c9ed220ce4bb79c08c08ca2199b07" member_type: VOTER }
I20250831 13:56:55.514097 19433 raft_consensus.cc:397] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250831 13:56:55.514369 19433 raft_consensus.cc:491] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250831 13:56:55.514669 19433 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:55.518370 19433 raft_consensus.cc:513] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525c9ed220ce4bb79c08c08ca2199b07" member_type: VOTER }
I20250831 13:56:55.519076 19433 leader_election.cc:304] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 525c9ed220ce4bb79c08c08ca2199b07; no voters:
I20250831 13:56:55.520928 19433 leader_election.cc:290] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250831 13:56:55.521157 19436 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:55.523932 19436 raft_consensus.cc:695] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 1 LEADER]: Becoming Leader. State: Replica: 525c9ed220ce4bb79c08c08ca2199b07, State: Running, Role: LEADER
I20250831 13:56:55.524813 19436 consensus_queue.cc:237] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525c9ed220ce4bb79c08c08ca2199b07" member_type: VOTER }
I20250831 13:56:55.525907 19433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 13:56:55.543041 19438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "525c9ed220ce4bb79c08c08ca2199b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525c9ed220ce4bb79c08c08ca2199b07" member_type: VOTER } }
I20250831 13:56:55.543273 19437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 525c9ed220ce4bb79c08c08ca2199b07. Latest consensus state: current_term: 1 leader_uuid: "525c9ed220ce4bb79c08c08ca2199b07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "525c9ed220ce4bb79c08c08ca2199b07" member_type: VOTER } }
I20250831 13:56:55.543721 19438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [sys.catalog]: This master's current role is: LEADER
I20250831 13:56:55.543921 19437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [sys.catalog]: This master's current role is: LEADER
I20250831 13:56:55.546866 19445 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 13:56:55.558193 19445 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 13:56:55.559959 19366 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250831 13:56:55.573228 19445 catalog_manager.cc:1349] Generated new cluster ID: 1fa37cda10ac4f8ab7f746ab80beaa53
I20250831 13:56:55.573513 19445 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 13:56:55.592052 19445 catalog_manager.cc:1372] Generated new certificate authority record
I20250831 13:56:55.594048 19445 catalog_manager.cc:1506] Loading token signing keys...
I20250831 13:56:55.609436 19445 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07: Generated new TSK 0
I20250831 13:56:55.610486 19445 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 13:56:55.626191 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:56:55.634464 19455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:56:55.635757 19456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:56:55.641680 19458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:56:55.643776 19366 server_base.cc:1047] running on GCE node
I20250831 13:56:55.644706 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:56:55.644976 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:56:55.645187 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648615645170 us; error 0 us; skew 500 ppm
I20250831 13:56:55.645970 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:56:55.650126 19366 webserver.cc:480] Webserver started at http://127.18.233.129:34829/ using document root <none> and password file <none>
I20250831 13:56:55.650859 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:56:55.651129 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:56:55.651468 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:56:55.652959 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-0-root/instance:
uuid: "6efac5f24fe647729a762108877cab9b"
format_stamp: "Formatted at 2025-08-31 13:56:55 on dist-test-slave-sg9n"
I20250831 13:56:55.659694 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250831 13:56:55.664631 19463 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:55.665446 19366 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250831 13:56:55.665783 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-0-root
uuid: "6efac5f24fe647729a762108877cab9b"
format_stamp: "Formatted at 2025-08-31 13:56:55 on dist-test-slave-sg9n"
I20250831 13:56:55.666177 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:56:55.699569 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:56:55.701344 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:56:55.704154 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:56:55.709507 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:56:55.709821 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:55.710150 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:56:55.710398 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:55.796869 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.129:38233
I20250831 13:56:55.796929 19525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.129:38233 every 8 connection(s)
I20250831 13:56:55.804395 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:56:55.814461 19530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:56:55.827250 19531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:56:55.830498 19366 server_base.cc:1047] running on GCE node
W20250831 13:56:55.830561 19533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:56:55.832015 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:56:55.832273 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:56:55.832479 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648615832463 us; error 0 us; skew 500 ppm
I20250831 13:56:55.833271 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:56:55.835057 19526 heartbeater.cc:344] Connected to a master server at 127.18.233.190:40083
I20250831 13:56:55.835460 19526 heartbeater.cc:461] Registering TS with master...
I20250831 13:56:55.836268 19366 webserver.cc:480] Webserver started at http://127.18.233.130:36061/ using document root <none> and password file <none>
I20250831 13:56:55.836383 19526 heartbeater.cc:507] Master 127.18.233.190:40083 requested a full tablet report, sending...
I20250831 13:56:55.836959 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:56:55.837208 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:56:55.837519 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:56:55.838927 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-1-root/instance:
uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
format_stamp: "Formatted at 2025-08-31 13:56:55 on dist-test-slave-sg9n"
I20250831 13:56:55.839103 19398 ts_manager.cc:194] Registered new tserver with Master: 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233)
I20250831 13:56:55.841153 19398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43538
I20250831 13:56:55.845492 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250831 13:56:55.850539 19538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:55.851446 19366 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250831 13:56:55.851754 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-1-root
uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
format_stamp: "Formatted at 2025-08-31 13:56:55 on dist-test-slave-sg9n"
I20250831 13:56:55.852108 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:56:55.869951 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:56:55.871505 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:56:55.873859 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:56:55.877314 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:56:55.877584 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:55.877877 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:56:55.878067 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:55.962971 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.130:35797
I20250831 13:56:55.963047 19600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.130:35797 every 8 connection(s)
I20250831 13:56:55.971473 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:56:55.979399 19604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:56:55.984757 19605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:56:55.985945 19601 heartbeater.cc:344] Connected to a master server at 127.18.233.190:40083
I20250831 13:56:55.986456 19601 heartbeater.cc:461] Registering TS with master...
I20250831 13:56:55.987637 19601 heartbeater.cc:507] Master 127.18.233.190:40083 requested a full tablet report, sending...
I20250831 13:56:55.990374 19398 ts_manager.cc:194] Registered new tserver with Master: c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:55.991747 19398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43540
W20250831 13:56:55.992918 19607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:56:55.996455 19366 server_base.cc:1047] running on GCE node
I20250831 13:56:55.997221 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:56:55.997390 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:56:55.997543 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648615997528 us; error 0 us; skew 500 ppm
I20250831 13:56:55.998073 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:56:56.000469 19366 webserver.cc:480] Webserver started at http://127.18.233.131:37607/ using document root <none> and password file <none>
I20250831 13:56:56.000923 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:56:56.001122 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:56:56.001353 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:56:56.002339 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-2-root/instance:
uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
format_stamp: "Formatted at 2025-08-31 13:56:55 on dist-test-slave-sg9n"
I20250831 13:56:56.006785 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250831 13:56:56.009878 19612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:56.010617 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250831 13:56:56.010927 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-2-root
uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
format_stamp: "Formatted at 2025-08-31 13:56:55 on dist-test-slave-sg9n"
I20250831 13:56:56.011197 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1756648613964447-19366-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:56:56.028234 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:56:56.029417 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:56:56.031417 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:56:56.034016 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:56:56.034216 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:56.034446 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:56:56.034598 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:56:56.119446 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.131:41591
I20250831 13:56:56.119616 19674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.131:41591 every 8 connection(s)
I20250831 13:56:56.135202 19675 heartbeater.cc:344] Connected to a master server at 127.18.233.190:40083
I20250831 13:56:56.135532 19675 heartbeater.cc:461] Registering TS with master...
I20250831 13:56:56.136232 19675 heartbeater.cc:507] Master 127.18.233.190:40083 requested a full tablet report, sending...
I20250831 13:56:56.137952 19398 ts_manager.cc:194] Registered new tserver with Master: 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:56.138536 19366 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014418666s
I20250831 13:56:56.139299 19398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43556
I20250831 13:56:56.142382 19366 test_util.cc:276] Using random seed: -1597856244
I20250831 13:56:56.188303 19398 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43572:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250831 13:56:56.191932 19398 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 13:56:56.288022 19635 tablet_service.cc:1468] Processing CreateTablet for tablet ac5c9cea365d40299758f1dbcfa1d5bb (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250831 13:56:56.288661 19636 tablet_service.cc:1468] Processing CreateTablet for tablet 1c5f9bcfbd354d6b8fc27a930266a723 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250831 13:56:56.289752 19635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c9cea365d40299758f1dbcfa1d5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.291044 19637 tablet_service.cc:1468] Processing CreateTablet for tablet ff828664880747f0831d670d6befb592 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250831 13:56:56.291887 19562 tablet_service.cc:1468] Processing CreateTablet for tablet 1c5f9bcfbd354d6b8fc27a930266a723 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250831 13:56:56.292425 19637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff828664880747f0831d670d6befb592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.293195 19562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5f9bcfbd354d6b8fc27a930266a723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.293967 19558 tablet_service.cc:1468] Processing CreateTablet for tablet 1bfadbd653f845489dcce6b6b819e776 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250831 13:56:56.295197 19558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bfadbd653f845489dcce6b6b819e776. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.296049 19560 tablet_service.cc:1468] Processing CreateTablet for tablet f934026fa41446ac9c7ad9aa51e2fa3d (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250831 13:56:56.297286 19560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f934026fa41446ac9c7ad9aa51e2fa3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.298429 19559 tablet_service.cc:1468] Processing CreateTablet for tablet 1bfa452f0b4b40b09a0e8adee0a0afbe (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250831 13:56:56.299760 19559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bfa452f0b4b40b09a0e8adee0a0afbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.300047 19566 tablet_service.cc:1468] Processing CreateTablet for tablet cdb3b7c54a244d82b47d49bbbd6f5f08 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250831 13:56:56.300081 19561 tablet_service.cc:1468] Processing CreateTablet for tablet ac5c9cea365d40299758f1dbcfa1d5bb (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250831 13:56:56.301436 19566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb3b7c54a244d82b47d49bbbd6f5f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.301162 19640 tablet_service.cc:1468] Processing CreateTablet for tablet cdb3b7c54a244d82b47d49bbbd6f5f08 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250831 13:56:56.302405 19640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb3b7c54a244d82b47d49bbbd6f5f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.303292 19561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c9cea365d40299758f1dbcfa1d5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.304005 19564 tablet_service.cc:1468] Processing CreateTablet for tablet bab0ce4153b84bf59b45f7788ff29c5f (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250831 13:56:56.304407 19638 tablet_service.cc:1468] Processing CreateTablet for tablet bab0ce4153b84bf59b45f7788ff29c5f (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250831 13:56:56.305297 19564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab0ce4153b84bf59b45f7788ff29c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.307904 19563 tablet_service.cc:1468] Processing CreateTablet for tablet ff828664880747f0831d670d6befb592 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250831 13:56:56.309242 19563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff828664880747f0831d670d6befb592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.308993 19634 tablet_service.cc:1468] Processing CreateTablet for tablet f934026fa41446ac9c7ad9aa51e2fa3d (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250831 13:56:56.320199 19565 tablet_service.cc:1468] Processing CreateTablet for tablet 74df8cf8627043ca9caa1c1b7c441fb3 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250831 13:56:56.322930 19638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab0ce4153b84bf59b45f7788ff29c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.315941 19633 tablet_service.cc:1468] Processing CreateTablet for tablet 1bfa452f0b4b40b09a0e8adee0a0afbe (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250831 13:56:56.325958 19639 tablet_service.cc:1468] Processing CreateTablet for tablet 74df8cf8627043ca9caa1c1b7c441fb3 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250831 13:56:56.328255 19565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74df8cf8627043ca9caa1c1b7c441fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.334959 19634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f934026fa41446ac9c7ad9aa51e2fa3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.329941 19632 tablet_service.cc:1468] Processing CreateTablet for tablet 1bfadbd653f845489dcce6b6b819e776 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250831 13:56:56.343029 19633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bfa452f0b4b40b09a0e8adee0a0afbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.346025 19636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5f9bcfbd354d6b8fc27a930266a723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.394436 19693 tablet_bootstrap.cc:492] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:56.397179 19639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74df8cf8627043ca9caa1c1b7c441fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.399328 19632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bfadbd653f845489dcce6b6b819e776. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.443025 19488 tablet_service.cc:1468] Processing CreateTablet for tablet ff828664880747f0831d670d6befb592 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20250831 13:56:56.445438 19488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff828664880747f0831d670d6befb592. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.448920 19491 tablet_service.cc:1468] Processing CreateTablet for tablet cdb3b7c54a244d82b47d49bbbd6f5f08 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION VALUES < 238609294
I20250831 13:56:56.452365 19490 tablet_service.cc:1468] Processing CreateTablet for tablet 74df8cf8627043ca9caa1c1b7c441fb3 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20250831 13:56:56.458007 19489 tablet_service.cc:1468] Processing CreateTablet for tablet bab0ce4153b84bf59b45f7788ff29c5f (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20250831 13:56:56.472141 19491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb3b7c54a244d82b47d49bbbd6f5f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.458205 19487 tablet_service.cc:1468] Processing CreateTablet for tablet 1c5f9bcfbd354d6b8fc27a930266a723 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20250831 13:56:56.477953 19489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bab0ce4153b84bf59b45f7788ff29c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.487272 19485 tablet_service.cc:1468] Processing CreateTablet for tablet f934026fa41446ac9c7ad9aa51e2fa3d (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20250831 13:56:56.468739 19486 tablet_service.cc:1468] Processing CreateTablet for tablet ac5c9cea365d40299758f1dbcfa1d5bb (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20250831 13:56:56.489452 19490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74df8cf8627043ca9caa1c1b7c441fb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.493192 19484 tablet_service.cc:1468] Processing CreateTablet for tablet 1bfa452f0b4b40b09a0e8adee0a0afbe (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20250831 13:56:56.497332 19486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c9cea365d40299758f1dbcfa1d5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.500237 19487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5f9bcfbd354d6b8fc27a930266a723. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.498793 19483 tablet_service.cc:1468] Processing CreateTablet for tablet 1bfadbd653f845489dcce6b6b819e776 (DEFAULT_TABLE table=test-workload [id=bbb80dd1c361410d8a1a5a152234a7e4]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20250831 13:56:56.519047 19484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bfa452f0b4b40b09a0e8adee0a0afbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.574456 19693 tablet_bootstrap.cc:654] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
W20250831 13:56:56.576330 19446 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:56:56.610631 19483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bfadbd653f845489dcce6b6b819e776. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.612450 19485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f934026fa41446ac9c7ad9aa51e2fa3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:56:56.670871 19693 tablet_bootstrap.cc:492] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:56.671391 19693 ts_tablet_manager.cc:1397] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.291s user 0.013s sys 0.007s
I20250831 13:56:56.698231 19693 raft_consensus.cc:357] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.703647 19693 raft_consensus.cc:383] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.712273 19696 tablet_bootstrap.cc:492] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:56.719249 19695 tablet_bootstrap.cc:492] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:56.711759 19693 raft_consensus.cc:738] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.731743 19693 consensus_queue.cc:260] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.735579 19695 tablet_bootstrap.cc:654] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.745541 19696 tablet_bootstrap.cc:654] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.756294 19693 ts_tablet_manager.cc:1428] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.085s user 0.011s sys 0.014s
I20250831 13:56:56.756211 19675 heartbeater.cc:499] Master 127.18.233.190:40083 was elected leader, sending a full tablet report...
I20250831 13:56:56.757330 19693 tablet_bootstrap.cc:492] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:56.767764 19693 tablet_bootstrap.cc:654] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.772473 19695 tablet_bootstrap.cc:492] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:56.772933 19695 ts_tablet_manager.cc:1397] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.016s
I20250831 13:56:56.775651 19695 raft_consensus.cc:357] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.776358 19695 raft_consensus.cc:383] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.776650 19695 raft_consensus.cc:738] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.777305 19695 consensus_queue.cc:260] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.792583 19696 tablet_bootstrap.cc:492] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:56.792631 19601 heartbeater.cc:499] Master 127.18.233.190:40083 was elected leader, sending a full tablet report...
I20250831 13:56:56.793030 19696 ts_tablet_manager.cc:1397] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.004s
I20250831 13:56:56.793352 19695 ts_tablet_manager.cc:1428] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
I20250831 13:56:56.794339 19695 tablet_bootstrap.cc:492] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:56.795217 19696 raft_consensus.cc:357] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:56.796728 19693 tablet_bootstrap.cc:492] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:56.797201 19693 ts_tablet_manager.cc:1397] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.017s
I20250831 13:56:56.800043 19693 raft_consensus.cc:357] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.800783 19695 tablet_bootstrap.cc:654] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.800855 19693 raft_consensus.cc:383] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.801359 19693 raft_consensus.cc:738] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.802211 19693 consensus_queue.cc:260] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.806370 19693 ts_tablet_manager.cc:1428] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250831 13:56:56.795900 19696 raft_consensus.cc:383] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.806828 19696 raft_consensus.cc:738] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.807389 19693 tablet_bootstrap.cc:492] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:56.811961 19695 tablet_bootstrap.cc:492] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:56.812384 19695 ts_tablet_manager.cc:1397] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20250831 13:56:56.814055 19693 tablet_bootstrap.cc:654] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.815040 19695 raft_consensus.cc:357] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.815712 19695 raft_consensus.cc:383] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.815966 19695 raft_consensus.cc:738] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.816604 19695 consensus_queue.cc:260] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.816736 19696 consensus_queue.cc:260] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:56.819396 19695 ts_tablet_manager.cc:1428] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 13:56:56.820340 19695 tablet_bootstrap.cc:492] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:56.826361 19695 tablet_bootstrap.cc:654] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.829435 19526 heartbeater.cc:499] Master 127.18.233.190:40083 was elected leader, sending a full tablet report...
I20250831 13:56:56.830202 19693 tablet_bootstrap.cc:492] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:56.830734 19693 ts_tablet_manager.cc:1397] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.002s
I20250831 13:56:56.833478 19693 raft_consensus.cc:357] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.834224 19693 raft_consensus.cc:383] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.834594 19696 ts_tablet_manager.cc:1428] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.041s user 0.007s sys 0.000s
I20250831 13:56:56.834539 19693 raft_consensus.cc:738] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.835676 19696 tablet_bootstrap.cc:492] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:56.835412 19693 consensus_queue.cc:260] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.837751 19693 ts_tablet_manager.cc:1428] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:56:56.838733 19693 tablet_bootstrap.cc:492] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:56.845240 19693 tablet_bootstrap.cc:654] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.845523 19696 tablet_bootstrap.cc:654] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.858127 19693 tablet_bootstrap.cc:492] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:56.858644 19693 ts_tablet_manager.cc:1397] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.003s
I20250831 13:56:56.860023 19695 tablet_bootstrap.cc:492] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:56.860467 19695 ts_tablet_manager.cc:1397] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.004s
I20250831 13:56:56.860461 19696 tablet_bootstrap.cc:492] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:56.861119 19696 ts_tablet_manager.cc:1397] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.005s
I20250831 13:56:56.861722 19693 raft_consensus.cc:357] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.862538 19693 raft_consensus.cc:383] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.862923 19693 raft_consensus.cc:738] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.863314 19695 raft_consensus.cc:357] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.864143 19695 raft_consensus.cc:383] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.863739 19696 raft_consensus.cc:357] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.864450 19695 raft_consensus.cc:738] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.863737 19693 consensus_queue.cc:260] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.864714 19696 raft_consensus.cc:383] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.865149 19696 raft_consensus.cc:738] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.865768 19696 consensus_queue.cc:260] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.866858 19693 ts_tablet_manager.cc:1428] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250831 13:56:56.868155 19696 ts_tablet_manager.cc:1428] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20250831 13:56:56.865361 19695 consensus_queue.cc:260] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.868963 19696 tablet_bootstrap.cc:492] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:56.867735 19693 tablet_bootstrap.cc:492] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:56.874917 19697 raft_consensus.cc:491] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:56.875384 19697 raft_consensus.cc:513] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.875963 19696 tablet_bootstrap.cc:654] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.878155 19697 leader_election.cc:290] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
W20250831 13:56:56.879824 19676 tablet.cc:2378] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 13:56:56.880299 19693 tablet_bootstrap.cc:654] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.885542 19695 ts_tablet_manager.cc:1428] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.025s user 0.004s sys 0.003s
I20250831 13:56:56.894639 19695 tablet_bootstrap.cc:492] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:56.903159 19695 tablet_bootstrap.cc:654] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.916216 19695 tablet_bootstrap.cc:492] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:56.916805 19695 ts_tablet_manager.cc:1397] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.015s
I20250831 13:56:56.920292 19696 tablet_bootstrap.cc:492] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:56.921226 19696 ts_tablet_manager.cc:1397] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.005s
I20250831 13:56:56.920360 19695 raft_consensus.cc:357] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.921785 19695 raft_consensus.cc:383] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.922168 19695 raft_consensus.cc:738] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.923193 19695 consensus_queue.cc:260] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.928189 19697 raft_consensus.cc:491] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:56.928643 19697 raft_consensus.cc:513] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.930457 19697 leader_election.cc:290] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:56.931514 19576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" is_pre_election: true
I20250831 13:56:56.932287 19576 raft_consensus.cc:2466] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 0.
I20250831 13:56:56.933120 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" is_pre_election: true
I20250831 13:56:56.934912 19695 ts_tablet_manager.cc:1428] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.018s user 0.008s sys 0.003s
I20250831 13:56:56.934908 19615 leader_election.cc:304] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:56.936254 19697 raft_consensus.cc:2802] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250831 13:56:56.936417 19615 leader_election.cc:343] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:56.936618 19697 raft_consensus.cc:491] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:56.936977 19697 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:56.939396 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:56.941383 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:56.941948 19501 raft_consensus.cc:2466] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 0.
I20250831 13:56:56.943749 19697 raft_consensus.cc:513] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.944196 19695 tablet_bootstrap.cc:492] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:56.945628 19697 leader_election.cc:290] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:56.946416 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
W20250831 13:56:56.947564 19615 leader_election.cc:343] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:56.948477 19615 leader_election.cc:304] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters: c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:56:56.950469 19697 raft_consensus.cc:2802] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:56.950840 19697 raft_consensus.cc:491] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:56.927152 19696 raft_consensus.cc:357] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.951264 19697 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:56.951457 19696 raft_consensus.cc:383] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.951781 19696 raft_consensus.cc:738] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.952508 19696 consensus_queue.cc:260] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.953377 19695 tablet_bootstrap.cc:654] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
W20250831 13:56:56.950799 19615 leader_election.cc:343] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:56.956043 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:56:56.956593 19575 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:56.957921 19697 raft_consensus.cc:513] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.959554 19693 tablet_bootstrap.cc:492] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:56.960235 19693 ts_tablet_manager.cc:1397] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.093s user 0.039s sys 0.041s
I20250831 13:56:56.963546 19575 raft_consensus.cc:2466] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 1.
I20250831 13:56:56.960917 19696 ts_tablet_manager.cc:1428] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.039s user 0.008s sys 0.000s
I20250831 13:56:56.964612 19615 leader_election.cc:304] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters: 6efac5f24fe647729a762108877cab9b
I20250831 13:56:56.965049 19696 tablet_bootstrap.cc:492] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:56.971628 19715 raft_consensus.cc:2802] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:56.973016 19693 raft_consensus.cc:357] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.974102 19693 raft_consensus.cc:383] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.974532 19693 raft_consensus.cc:738] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.975569 19693 consensus_queue.cc:260] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.976509 19715 raft_consensus.cc:695] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:56.977394 19715 consensus_queue.cc:237] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.980376 19696 tablet_bootstrap.cc:654] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.981597 19693 ts_tablet_manager.cc:1428] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20250831 13:56:56.983603 19693 tablet_bootstrap.cc:492] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:56.984369 19697 leader_election.cc:290] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:56.985879 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:56:56.985940 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:56.986589 19501 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:56.988489 19695 tablet_bootstrap.cc:492] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:56.988960 19695 ts_tablet_manager.cc:1397] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.045s user 0.017s sys 0.024s
W20250831 13:56:56.990964 19615 leader_election.cc:343] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:56.991853 19695 raft_consensus.cc:357] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.993050 19695 raft_consensus.cc:383] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:56.993341 19695 raft_consensus.cc:738] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:56.994140 19501 raft_consensus.cc:2466] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 1.
I20250831 13:56:56.994065 19693 tablet_bootstrap.cc:654] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:56.994083 19695 consensus_queue.cc:260] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:56.995319 19615 leader_election.cc:304] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters: c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:56:56.996061 19697 raft_consensus.cc:2802] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:56.996522 19697 raft_consensus.cc:695] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:56.997539 19697 consensus_queue.cc:237] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.003347 19696 tablet_bootstrap.cc:492] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:57.003892 19696 ts_tablet_manager.cc:1397] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.039s user 0.017s sys 0.000s
I20250831 13:56:57.004268 19695 ts_tablet_manager.cc:1428] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20250831 13:56:57.006856 19696 raft_consensus.cc:357] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.007607 19696 raft_consensus.cc:383] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.007997 19696 raft_consensus.cc:738] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.008735 19696 consensus_queue.cc:260] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.009634 19695 tablet_bootstrap.cc:492] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:57.009927 19396 catalog_manager.cc:5582] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:56:57.011489 19696 ts_tablet_manager.cc:1428] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 13:56:57.012419 19696 tablet_bootstrap.cc:492] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:57.020990 19695 tablet_bootstrap.cc:654] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.021376 19693 tablet_bootstrap.cc:492] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:57.021893 19693 ts_tablet_manager.cc:1397] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.012s
I20250831 13:56:57.024681 19693 raft_consensus.cc:357] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.025440 19693 raft_consensus.cc:383] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.025861 19693 raft_consensus.cc:738] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.026767 19693 consensus_queue.cc:260] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.029201 19693 ts_tablet_manager.cc:1428] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:56:57.029455 19696 tablet_bootstrap.cc:654] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.030197 19693 tablet_bootstrap.cc:492] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:57.037066 19693 tablet_bootstrap.cc:654] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.038471 19695 tablet_bootstrap.cc:492] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:57.039083 19695 ts_tablet_manager.cc:1397] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.005s
I20250831 13:56:57.042558 19695 raft_consensus.cc:357] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.043473 19695 raft_consensus.cc:383] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.043828 19695 raft_consensus.cc:738] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.044950 19695 consensus_queue.cc:260] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.046912 19697 raft_consensus.cc:491] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.047392 19697 raft_consensus.cc:513] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.049325 19697 leader_election.cc:290] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.050776 19695 ts_tablet_manager.cc:1428] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20250831 13:56:57.052002 19695 tablet_bootstrap.cc:492] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:57.052079 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.053531 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" is_pre_election: true
W20250831 13:56:57.059203 19615 leader_election.cc:343] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:56:57.060024 19615 leader_election.cc:343] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:57.060518 19615 leader_election.cc:304] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626; no voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:56:57.061437 19697 raft_consensus.cc:2747] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:56:57.045920 19396 catalog_manager.cc:5582] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:56:57.070060 19695 tablet_bootstrap.cc:654] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.081868 19696 tablet_bootstrap.cc:492] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:57.082487 19696 ts_tablet_manager.cc:1397] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.070s user 0.029s sys 0.024s
I20250831 13:56:57.086118 19696 raft_consensus.cc:357] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.087105 19696 raft_consensus.cc:383] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.087488 19696 raft_consensus.cc:738] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.088366 19696 consensus_queue.cc:260] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.092979 19693 tablet_bootstrap.cc:492] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:57.093447 19693 ts_tablet_manager.cc:1397] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.063s user 0.033s sys 0.016s
I20250831 13:56:57.094435 19696 ts_tablet_manager.cc:1428] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20250831 13:56:57.095381 19696 tablet_bootstrap.cc:492] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:57.096230 19693 raft_consensus.cc:357] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.096895 19693 raft_consensus.cc:383] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.097177 19693 raft_consensus.cc:738] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.097894 19693 consensus_queue.cc:260] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.093502 19695 tablet_bootstrap.cc:492] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:57.099334 19695 ts_tablet_manager.cc:1397] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.011s
I20250831 13:56:57.100360 19693 ts_tablet_manager.cc:1428] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 13:56:57.101279 19693 tablet_bootstrap.cc:492] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:57.101883 19695 raft_consensus.cc:357] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.106077 19695 raft_consensus.cc:383] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.106384 19695 raft_consensus.cc:738] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.106961 19697 raft_consensus.cc:491] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.107398 19697 raft_consensus.cc:513] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.108089 19693 tablet_bootstrap.cc:654] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.109333 19697 leader_election.cc:290] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.109968 19695 consensus_queue.cc:260] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.112668 19695 ts_tablet_manager.cc:1428] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250831 13:56:57.113657 19695 tablet_bootstrap.cc:492] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:57.114259 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" is_pre_election: true
I20250831 13:56:57.113570 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.116298 19575 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 0.
I20250831 13:56:57.117430 19615 leader_election.cc:304] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
W20250831 13:56:57.118243 19615 leader_election.cc:343] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:57.118185 19697 raft_consensus.cc:2802] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:57.118955 19697 raft_consensus.cc:491] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:57.119431 19697 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.105336 19696 tablet_bootstrap.cc:654] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.126716 19697 raft_consensus.cc:513] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.128690 19697 leader_election.cc:290] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.130084 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.131070 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
W20250831 13:56:57.131327 19615 leader_election.cc:343] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:57.131610 19575 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.134284 19693 tablet_bootstrap.cc:492] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:57.134733 19693 ts_tablet_manager.cc:1397] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.010s
I20250831 13:56:57.134959 19695 tablet_bootstrap.cc:654] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.137183 19693 raft_consensus.cc:357] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.137856 19693 raft_consensus.cc:383] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.137797 19575 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 1.
I20250831 13:56:57.138180 19693 raft_consensus.cc:738] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.138957 19615 leader_election.cc:304] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters: 6efac5f24fe647729a762108877cab9b
I20250831 13:56:57.138825 19693 consensus_queue.cc:260] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.139968 19716 raft_consensus.cc:491] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.139998 19715 raft_consensus.cc:2802] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:57.140841 19693 ts_tablet_manager.cc:1428] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:56:57.140564 19716 raft_consensus.cc:513] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.141798 19693 tablet_bootstrap.cc:492] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626: Bootstrap starting.
I20250831 13:56:57.142642 19716 leader_election.cc:290] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.143641 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.144275 19501 raft_consensus.cc:2466] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 0.
I20250831 13:56:57.145906 19576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" is_pre_election: true
I20250831 13:56:57.146512 19576 raft_consensus.cc:2466] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 0.
I20250831 13:56:57.147948 19615 leader_election.cc:304] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.148293 19693 tablet_bootstrap.cc:654] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.149230 19716 raft_consensus.cc:2802] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:57.149598 19716 raft_consensus.cc:491] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:57.149938 19716 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.160085 19693 tablet_bootstrap.cc:492] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626: No bootstrap required, opened a new log
I20250831 13:56:57.160625 19693 ts_tablet_manager.cc:1397] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.006s
I20250831 13:56:57.163448 19716 raft_consensus.cc:513] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.163571 19693 raft_consensus.cc:357] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.164163 19693 raft_consensus.cc:383] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.164413 19693 raft_consensus.cc:738] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.164528 19696 tablet_bootstrap.cc:492] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:57.165025 19696 ts_tablet_manager.cc:1397] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.006s
I20250831 13:56:57.165079 19693 consensus_queue.cc:260] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.166054 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.166582 19501 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.167652 19693 ts_tablet_manager.cc:1428] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250831 13:56:57.168341 19696 raft_consensus.cc:357] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.169037 19696 raft_consensus.cc:383] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.169312 19696 raft_consensus.cc:738] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.169965 19696 consensus_queue.cc:260] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.171701 19501 raft_consensus.cc:2466] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 1.
I20250831 13:56:57.172835 19615 leader_election.cc:304] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:56:57.173950 19697 raft_consensus.cc:2802] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:57.174404 19697 raft_consensus.cc:695] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.175413 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:56:57.175199 19697 consensus_queue.cc:237] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.175916 19575 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.176565 19701 raft_consensus.cc:491] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.177008 19701 raft_consensus.cc:513] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.178666 19696 ts_tablet_manager.cc:1428] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250831 13:56:57.179517 19701 leader_election.cc:290] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.183187 19716 leader_election.cc:290] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.187652 19715 raft_consensus.cc:695] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.187928 19696 tablet_bootstrap.cc:492] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:57.188719 19715 consensus_queue.cc:237] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.195943 19696 tablet_bootstrap.cc:654] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.200999 19575 raft_consensus.cc:2466] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 1.
I20250831 13:56:57.210260 19704 raft_consensus.cc:491] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.210777 19704 raft_consensus.cc:513] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.213447 19704 leader_election.cc:290] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.220678 19396 catalog_manager.cc:5582] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:56:57.236413 19701 raft_consensus.cc:491] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.236902 19701 raft_consensus.cc:513] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.237298 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfadbd653f845489dcce6b6b819e776" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" is_pre_election: true
I20250831 13:56:57.238296 19650 raft_consensus.cc:2466] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 0.
I20250831 13:56:57.240089 19540 leader_election.cc:304] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.241902 19396 catalog_manager.cc:5582] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.244252 19701 leader_election.cc:290] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.244762 19701 raft_consensus.cc:2802] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:57.245234 19701 raft_consensus.cc:491] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:57.245678 19701 raft_consensus.cc:3058] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.254603 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f934026fa41446ac9c7ad9aa51e2fa3d" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" is_pre_election: true
I20250831 13:56:57.255271 19650 raft_consensus.cc:2466] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 0.
I20250831 13:56:57.256281 19540 leader_election.cc:304] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.257249 19742 raft_consensus.cc:2802] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:57.257642 19742 raft_consensus.cc:491] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:57.257997 19742 raft_consensus.cc:3058] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.257741 19695 tablet_bootstrap.cc:492] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:57.259171 19695 ts_tablet_manager.cc:1397] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.146s user 0.014s sys 0.029s
I20250831 13:56:57.264595 19742 raft_consensus.cc:513] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.267077 19742 leader_election.cc:290] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.267968 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfadbd653f845489dcce6b6b819e776" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.268584 19501 raft_consensus.cc:2466] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 0.
I20250831 13:56:57.268525 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f934026fa41446ac9c7ad9aa51e2fa3d" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.268379 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f934026fa41446ac9c7ad9aa51e2fa3d" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.270563 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f934026fa41446ac9c7ad9aa51e2fa3d" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.270596 19650 raft_consensus.cc:3058] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
W20250831 13:56:57.271508 19541 leader_election.cc:343] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:56:57.272789 19541 leader_election.cc:343] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:57.273236 19696 tablet_bootstrap.cc:492] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:57.273777 19696 ts_tablet_manager.cc:1397] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.086s user 0.011s sys 0.003s
I20250831 13:56:57.277736 19650 raft_consensus.cc:2466] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 1.
I20250831 13:56:57.277662 19696 raft_consensus.cc:357] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.281648 19696 raft_consensus.cc:383] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.279060 19540 leader_election.cc:304] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters: 6efac5f24fe647729a762108877cab9b
I20250831 13:56:57.261623 19701 raft_consensus.cc:513] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.283354 19742 raft_consensus.cc:2802] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:57.284382 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfadbd653f845489dcce6b6b819e776" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.285079 19500 raft_consensus.cc:3058] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.286244 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfadbd653f845489dcce6b6b819e776" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.286796 19650 raft_consensus.cc:3058] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.290386 19500 raft_consensus.cc:2466] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 1.
I20250831 13:56:57.285313 19701 leader_election.cc:290] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.281010 19695 raft_consensus.cc:357] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.290963 19695 raft_consensus.cc:383] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.291301 19695 raft_consensus.cc:738] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.282014 19696 raft_consensus.cc:738] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.292191 19695 consensus_queue.cc:260] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.291805 19541 leader_election.cc:304] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.291863 19742 raft_consensus.cc:695] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:56:57.293929 19701 raft_consensus.cc:2802] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:57.294458 19701 raft_consensus.cc:695] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:56:57.296149 19695 ts_tablet_manager.cc:1428] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.034s user 0.002s sys 0.005s
I20250831 13:56:57.294376 19742 consensus_queue.cc:237] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.297326 19695 tablet_bootstrap.cc:492] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0: Bootstrap starting.
I20250831 13:56:57.297857 19701 consensus_queue.cc:237] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.297204 19696 consensus_queue.cc:260] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.303694 19650 raft_consensus.cc:2466] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 1.
I20250831 13:56:57.308346 19695 tablet_bootstrap.cc:654] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.312690 19696 ts_tablet_manager.cc:1428] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.039s user 0.007s sys 0.004s
I20250831 13:56:57.313796 19696 tablet_bootstrap.cc:492] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:57.317067 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" is_pre_election: true
I20250831 13:56:57.316393 19398 catalog_manager.cc:5582] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.317648 19650 raft_consensus.cc:2466] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 0.
I20250831 13:56:57.319185 19465 leader_election.cc:304] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:56:57.320132 19704 raft_consensus.cc:2802] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:57.320469 19704 raft_consensus.cc:491] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:57.320971 19704 raft_consensus.cc:3058] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.324093 19696 tablet_bootstrap.cc:654] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.329517 19704 raft_consensus.cc:513] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.331861 19704 leader_election.cc:290] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 1 election: Requested vote from peers c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.332834 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.333375 19650 raft_consensus.cc:3058] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.339608 19650 raft_consensus.cc:2466] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 1.
I20250831 13:56:57.340580 19465 leader_election.cc:304] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:56:57.344487 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" is_pre_election: true
I20250831 13:56:57.343809 19396 catalog_manager.cc:5582] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.345618 19576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
W20250831 13:56:57.347159 19466 leader_election.cc:343] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:56:57.347987 19466 leader_election.cc:343] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:56:57.349133 19742 raft_consensus.cc:491] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.349576 19742 raft_consensus.cc:513] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.351157 19704 raft_consensus.cc:2802] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:57.351389 19742 leader_election.cc:290] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.354759 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.355381 19500 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 0.
I20250831 13:56:57.356844 19704 raft_consensus.cc:695] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 1 LEADER]: Becoming Leader. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:56:57.356897 19541 leader_election.cc:304] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.357781 19742 raft_consensus.cc:2802] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:57.358152 19695 tablet_bootstrap.cc:492] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0: No bootstrap required, opened a new log
I20250831 13:56:57.358238 19742 raft_consensus.cc:491] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:57.358911 19695 ts_tablet_manager.cc:1397] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.016s
I20250831 13:56:57.358820 19742 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.357736 19704 consensus_queue.cc:237] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.359958 19697 raft_consensus.cc:491] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.360093 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" is_pre_election: true
I20250831 13:56:57.360402 19697 raft_consensus.cc:513] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.361357 19650 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 0.
I20250831 13:56:57.362231 19697 leader_election.cc:290] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.363695 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.363946 19576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" is_pre_election: true
I20250831 13:56:57.364280 19500 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 0.
I20250831 13:56:57.364219 19695 raft_consensus.cc:357] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.364864 19695 raft_consensus.cc:383] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.365150 19695 raft_consensus.cc:738] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.365746 19696 tablet_bootstrap.cc:492] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:57.365564 19615 leader_election.cc:304] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:56:57.366153 19696 ts_tablet_manager.cc:1397] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.009s
I20250831 13:56:57.366328 19697 raft_consensus.cc:2802] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:57.366748 19697 raft_consensus.cc:491] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:57.367193 19697 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.368777 19696 raft_consensus.cc:357] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.369418 19696 raft_consensus.cc:383] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.369659 19696 raft_consensus.cc:738] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.370258 19696 consensus_queue.cc:260] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.372210 19696 ts_tablet_manager.cc:1428] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250831 13:56:57.373114 19696 tablet_bootstrap.cc:492] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b: Bootstrap starting.
I20250831 13:56:57.373236 19697 raft_consensus.cc:513] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.375123 19697 leader_election.cc:290] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.365943 19695 consensus_queue.cc:260] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.376255 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:56:57.366564 19742 raft_consensus.cc:513] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.378924 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.379413 19576 raft_consensus.cc:2391] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ce6fc4a86d04c5388b840b1b18c3626 in current term 1: Already voted for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in this term.
I20250831 13:56:57.379560 19500 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.381110 19695 ts_tablet_manager.cc:1428] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20250831 13:56:57.383831 19742 leader_election.cc:290] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.384086 19696 tablet_bootstrap.cc:654] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b: Neither blocks nor log segments found. Creating new log.
I20250831 13:56:57.385046 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.385521 19742 raft_consensus.cc:491] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.385955 19742 raft_consensus.cc:513] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.387766 19742 leader_election.cc:290] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.388448 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfa452f0b4b40b09a0e8adee0a0afbe" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.389118 19501 raft_consensus.cc:2466] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 0.
I20250831 13:56:57.389108 19398 catalog_manager.cc:5582] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6efac5f24fe647729a762108877cab9b (127.18.233.129). New cstate: current_term: 1 leader_uuid: "6efac5f24fe647729a762108877cab9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.390745 19649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfa452f0b4b40b09a0e8adee0a0afbe" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" is_pre_election: true
I20250831 13:56:57.391443 19649 raft_consensus.cc:2466] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 0.
I20250831 13:56:57.392627 19540 leader_election.cc:304] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.393530 19742 raft_consensus.cc:2802] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:56:57.393919 19742 raft_consensus.cc:491] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:56:57.394280 19742 raft_consensus.cc:3058] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.401095 19742 raft_consensus.cc:513] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.404470 19501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfa452f0b4b40b09a0e8adee0a0afbe" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.405093 19501 raft_consensus.cc:3058] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.387337 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.411513 19650 raft_consensus.cc:2391] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c6bcc1bb41fa4645b81d2c10907edfa0 in current term 1: Already voted for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in this term.
I20250831 13:56:57.412592 19540 leader_election.cc:304] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6bcc1bb41fa4645b81d2c10907edfa0; no voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b
I20250831 13:56:57.413468 19701 raft_consensus.cc:2747] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 13:56:57.414973 19501 raft_consensus.cc:2466] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 1.
I20250831 13:56:57.415980 19541 leader_election.cc:304] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.416870 19701 raft_consensus.cc:2802] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:57.417369 19701 raft_consensus.cc:695] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:56:57.418234 19701 consensus_queue.cc:237] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.420171 19500 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 1.
I20250831 13:56:57.422051 19615 leader_election.cc:304] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters: c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:56:57.422816 19697 raft_consensus.cc:2802] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:56:57.423296 19697 raft_consensus.cc:695] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.424265 19649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfa452f0b4b40b09a0e8adee0a0afbe" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.424029 19697 consensus_queue.cc:237] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.424782 19649 raft_consensus.cc:3058] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.430841 19696 tablet_bootstrap.cc:492] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b: No bootstrap required, opened a new log
I20250831 13:56:57.431265 19696 ts_tablet_manager.cc:1397] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b: Time spent bootstrapping tablet: real 0.058s user 0.019s sys 0.023s
I20250831 13:56:57.431499 19649 raft_consensus.cc:2466] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 1.
I20250831 13:56:57.432936 19742 leader_election.cc:290] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.432631 19396 catalog_manager.cc:5582] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:56:57.433523 19696 raft_consensus.cc:357] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.434159 19696 raft_consensus.cc:383] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:56:57.434378 19696 raft_consensus.cc:738] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Initialized, Role: FOLLOWER
I20250831 13:56:57.434983 19696 consensus_queue.cc:260] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.437083 19696 ts_tablet_manager.cc:1428] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:56:57.445191 19397 catalog_manager.cc:5582] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.468227 19704 raft_consensus.cc:491] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.468683 19704 raft_consensus.cc:513] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.469053 19701 raft_consensus.cc:491] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:56:57.470628 19704 leader_election.cc:290] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.471303 19576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" is_pre_election: true
I20250831 13:56:57.472044 19576 raft_consensus.cc:2391] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6efac5f24fe647729a762108877cab9b in current term 1: Already voted for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in this term.
I20250831 13:56:57.469694 19701 raft_consensus.cc:513] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.473132 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" is_pre_election: true
I20250831 13:56:57.474730 19465 leader_election.cc:304] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6efac5f24fe647729a762108877cab9b; no voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:56:57.475485 19704 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.476295 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efac5f24fe647729a762108877cab9b" is_pre_election: true
I20250831 13:56:57.477087 19500 raft_consensus.cc:2391] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6bcc1bb41fa4645b81d2c10907edfa0 in current term 1: Already voted for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in this term.
I20250831 13:56:57.476768 19697 consensus_queue.cc:1035] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.478070 19576 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.479208 19701 leader_election.cc:290] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.482414 19704 raft_consensus.cc:2747] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:56:57.490097 19649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" is_pre_election: true
I20250831 13:56:57.490433 19697 consensus_queue.cc:1035] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:56:57.491492 19540 leader_election.cc:304] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6bcc1bb41fa4645b81d2c10907edfa0; no voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b
I20250831 13:56:57.492471 19701 raft_consensus.cc:2747] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:56:57.496009 19650 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
mode: GRACEFUL
new_leader_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:35222
I20250831 13:56:57.496658 19650 raft_consensus.cc:604] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Received request to transfer leadership to TS 6efac5f24fe647729a762108877cab9b
I20250831 13:56:57.512192 19650 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
mode: GRACEFUL
new_leader_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:35222
I20250831 13:56:57.513199 19650 raft_consensus.cc:604] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Received request to transfer leadership to TS 6efac5f24fe647729a762108877cab9b
I20250831 13:56:57.518543 19650 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ff828664880747f0831d670d6befb592"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
mode: GRACEFUL
new_leader_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:35222
I20250831 13:56:57.531479 19650 raft_consensus.cc:604] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Received request to transfer leadership to TS 6efac5f24fe647729a762108877cab9b
I20250831 13:56:57.538573 19650 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
mode: GRACEFUL
new_leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:35222
I20250831 13:56:57.539146 19650 raft_consensus.cc:604] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Received request to transfer leadership to TS c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:56:57.545538 19650 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
mode: GRACEFUL
new_leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:35222
I20250831 13:56:57.546080 19650 raft_consensus.cc:604] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Received request to transfer leadership to TS c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:56:57.550478 19697 raft_consensus.cc:991] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626: : Instructing follower 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:56:57.550887 19501 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.550922 19697 raft_consensus.cc:1079] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Signalling peer 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:56:57.552688 19500 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ff828664880747f0831d670d6befb592"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:59350
I20250831 13:56:57.553586 19500 raft_consensus.cc:491] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:56:57.553968 19500 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.560904 19731 consensus_queue.cc:1035] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:56:57.563823 19500 raft_consensus.cc:513] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.566915 19500 leader_election.cc:290] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 2 election: Requested vote from peers c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.567142 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.567442 19576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:56:57.567734 19650 raft_consensus.cc:3053] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:56:57.567991 19576 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.568075 19650 raft_consensus.cc:738] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.568899 19650 consensus_queue.cc:260] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.570264 19650 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.576426 19650 raft_consensus.cc:2466] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 2.
I20250831 13:56:57.577477 19465 leader_election.cc:304] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:56:57.577674 19697 consensus_queue.cc:1035] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.578317 19704 raft_consensus.cc:2802] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:56:57.578825 19704 raft_consensus.cc:695] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Becoming Leader. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:56:57.578944 19697 consensus_queue.cc:1035] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.579695 19704 consensus_queue.cc:237] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.594424 19396 catalog_manager.cc:5582] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b reported cstate change: term changed from 1 to 2, leader changed from 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131) to 6efac5f24fe647729a762108877cab9b (127.18.233.129). New cstate: current_term: 2 leader_uuid: "6efac5f24fe647729a762108877cab9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.603593 19574 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1bfadbd653f845489dcce6b6b819e776"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
mode: GRACEFUL
new_leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:36214
I20250831 13:56:57.604318 19574 raft_consensus.cc:604] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Received request to transfer leadership to TS 5ce6fc4a86d04c5388b840b1b18c3626
I20250831 13:56:57.607669 19576 raft_consensus.cc:2466] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 2.
I20250831 13:56:57.625241 19715 raft_consensus.cc:991] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626: : Instructing follower c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:56:57.625665 19715 raft_consensus.cc:1079] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Signalling peer c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:56:57.628322 19574 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:36196
I20250831 13:56:57.629154 19574 raft_consensus.cc:491] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:56:57.629508 19574 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.633682 19697 raft_consensus.cc:991] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626: : Instructing follower 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:56:57.634064 19697 raft_consensus.cc:1079] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Signalling peer 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:56:57.636379 19501 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:59350
I20250831 13:56:57.636935 19501 raft_consensus.cc:491] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:56:57.637279 19501 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.638343 19715 consensus_queue.cc:1035] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20250831 13:56:57.640868 19697 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250831 13:56:57.637176 19574 raft_consensus.cc:513] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.651369 19574 leader_election.cc:290] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 2 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.652447 19498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.653275 19498 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.656129 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.656852 19650 raft_consensus.cc:3053] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:56:57.657253 19650 raft_consensus.cc:738] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.658213 19650 consensus_queue.cc:260] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.666302 19498 raft_consensus.cc:2466] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 2.
I20250831 13:56:57.666508 19650 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.667505 19541 leader_election.cc:304] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.668699 19701 raft_consensus.cc:2802] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:56:57.669273 19701 raft_consensus.cc:695] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:56:57.670538 19501 raft_consensus.cc:513] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.670286 19701 consensus_queue.cc:237] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.673207 19500 raft_consensus.cc:1238] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Rejecting Update request from peer 5ce6fc4a86d04c5388b840b1b18c3626 for earlier term 1. Current term is 2. Ops: []
I20250831 13:56:57.677367 19501 leader_election.cc:290] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 2 election: Requested vote from peers 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.681350 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:56:57.681625 19649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.681869 19575 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.682200 19649 raft_consensus.cc:3053] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:56:57.682538 19649 raft_consensus.cc:738] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.683265 19649 consensus_queue.cc:260] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.684464 19649 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.683732 19501 raft_consensus.cc:3058] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.688472 19575 raft_consensus.cc:2466] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 2.
I20250831 13:56:57.688946 19650 raft_consensus.cc:2466] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 2.
I20250831 13:56:57.689525 19466 leader_election.cc:304] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.690101 19697 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.690409 19704 raft_consensus.cc:2802] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:56:57.690868 19704 raft_consensus.cc:695] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Becoming Leader. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:56:57.691644 19704 consensus_queue.cc:237] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:56:57.701550 19742 consensus_queue.cc:1035] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.701736 19398 catalog_manager.cc:5582] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 1 to 2, leader changed from 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131) to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.714221 19716 raft_consensus.cc:991] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626: : Instructing follower c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:56:57.714695 19715 raft_consensus.cc:1079] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Signalling peer c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:56:57.716359 19575 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:36196
I20250831 13:56:57.716900 19575 raft_consensus.cc:491] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:56:57.717253 19575 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.723089 19743 consensus_queue.cc:1035] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.727558 19575 raft_consensus.cc:513] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.728648 19574 raft_consensus.cc:1238] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Rejecting Update request from peer 5ce6fc4a86d04c5388b840b1b18c3626 for earlier term 1. Current term is 2. Ops: []
I20250831 13:56:57.731356 19397 catalog_manager.cc:5582] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b reported cstate change: term changed from 1 to 2, leader changed from 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131) to 6efac5f24fe647729a762108877cab9b (127.18.233.129). New cstate: current_term: 2 leader_uuid: "6efac5f24fe647729a762108877cab9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.732640 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.733464 19575 leader_election.cc:290] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 2 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.733902 19648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.734480 19648 raft_consensus.cc:3053] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:56:57.734793 19648 raft_consensus.cc:738] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.735512 19648 consensus_queue.cc:260] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.736739 19648 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.742777 19499 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.744540 19648 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 2.
I20250831 13:56:57.745708 19540 leader_election.cc:304] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.746562 19743 raft_consensus.cc:2802] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:56:57.748229 19743 raft_consensus.cc:695] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:56:57.749033 19743 consensus_queue.cc:237] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.749385 19499 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 2.
I20250831 13:56:57.750872 19649 raft_consensus.cc:2466] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 2.
I20250831 13:56:57.764564 19396 catalog_manager.cc:5582] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 1 to 2, leader changed from 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131) to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.784611 19743 consensus_queue.cc:1035] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.794278 19575 raft_consensus.cc:3058] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:56:57.795835 19743 consensus_queue.cc:1035] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.806203 19704 consensus_queue.cc:1035] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.818528 19704 consensus_queue.cc:1035] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.831346 19701 raft_consensus.cc:991] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0: : Instructing follower 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:56:57.831759 19701 raft_consensus.cc:1079] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Signalling peer 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:56:57.833596 19648 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1bfadbd653f845489dcce6b6b819e776"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:35206
I20250831 13:56:57.834307 19648 raft_consensus.cc:491] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:56:57.834767 19648 raft_consensus.cc:3058] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.838939 19743 consensus_queue.cc:1035] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.844969 19648 raft_consensus.cc:513] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.856426 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfadbd653f845489dcce6b6b819e776" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:56:57.858147 19701 consensus_queue.cc:1035] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.858958 19499 raft_consensus.cc:3058] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.865429 19499 raft_consensus.cc:2466] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 2.
I20250831 13:56:57.866581 19648 leader_election.cc:290] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 2 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:56:57.867182 19715 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.869210 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bfadbd653f845489dcce6b6b819e776" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:56:57.869850 19575 raft_consensus.cc:3053] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:56:57.870288 19575 raft_consensus.cc:738] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:56:57.871281 19575 consensus_queue.cc:260] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.872823 19575 raft_consensus.cc:3058] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.876883 19615 leader_election.cc:304] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:56:57.877938 19697 raft_consensus.cc:2802] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:56:57.878368 19697 raft_consensus.cc:695] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.879094 19697 consensus_queue.cc:237] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.882372 19731 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:56:57.892354 19396 catalog_manager.cc:5582] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 1 to 2, leader changed from c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130) to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 2 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:56:57.898303 19575 raft_consensus.cc:2466] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 2.
I20250831 13:56:57.907517 19715 raft_consensus.cc:991] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626: : Instructing follower 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:56:57.907933 19715 raft_consensus.cc:1079] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Signalling peer 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:56:57.917028 19499 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:59350
I20250831 13:56:57.917554 19499 raft_consensus.cc:491] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:56:57.917910 19499 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.923702 19499 raft_consensus.cc:513] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.925040 19499 leader_election.cc:290] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 2 election: Requested vote from peers c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:56:57.925910 19574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:56:57.926052 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:56:57.926448 19574 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.926609 19650 raft_consensus.cc:3053] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:56:57.926980 19650 raft_consensus.cc:738] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:56:57.927723 19650 consensus_queue.cc:260] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.928890 19650 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:56:57.930995 19574 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 2.
I20250831 13:56:57.932040 19466 leader_election.cc:304] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:56:57.932641 19704 raft_consensus.cc:2802] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:56:57.933034 19704 raft_consensus.cc:695] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Becoming Leader. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:56:57.933825 19704 consensus_queue.cc:237] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:56:57.935890 19650 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 2.
I20250831 13:56:57.940209 19397 catalog_manager.cc:5582] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b reported cstate change: term changed from 1 to 2, leader changed from 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131) to 6efac5f24fe647729a762108877cab9b (127.18.233.129). New cstate: current_term: 2 leader_uuid: "6efac5f24fe647729a762108877cab9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:56:57.961375 19574 raft_consensus.cc:1273] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:57.962601 19704 consensus_queue.cc:1035] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:57.971395 19650 raft_consensus.cc:1273] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:57.972770 19770 consensus_queue.cc:1035] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.065337 19499 raft_consensus.cc:1273] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.066574 19701 consensus_queue.cc:1035] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.074342 19648 raft_consensus.cc:1273] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.075546 19743 consensus_queue.cc:1035] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.148133 19574 raft_consensus.cc:1273] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.149264 19770 consensus_queue.cc:1035] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.158470 19648 raft_consensus.cc:1273] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.160200 19770 consensus_queue.cc:1035] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.214100 19499 raft_consensus.cc:1273] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.215245 19743 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.222673 19648 raft_consensus.cc:1273] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.224057 19743 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.325729 19574 raft_consensus.cc:1273] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.326974 19715 consensus_queue.cc:1035] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.335697 19499 raft_consensus.cc:1273] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.337592 19697 consensus_queue.cc:1035] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.342020 19648 raft_consensus.cc:1273] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.343393 19770 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:56:58.353778 19574 raft_consensus.cc:1273] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:56:58.355264 19770 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:00.631649 19366 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20250831 13:57:00.632463 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:00.646842 19574 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
mode: GRACEFUL
new_leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:36218
I20250831 13:57:00.647454 19574 raft_consensus.cc:604] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Received request to transfer leadership to TS 5ce6fc4a86d04c5388b840b1b18c3626
I20250831 13:57:00.658977 19499 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
mode: GRACEFUL
new_leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:59370
I20250831 13:57:00.659479 19499 raft_consensus.cc:604] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Received request to transfer leadership to TS 5ce6fc4a86d04c5388b840b1b18c3626
I20250831 13:57:00.660548 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250831 13:57:00.661070 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:00.948925 19701 raft_consensus.cc:991] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0: : Instructing follower 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:57:00.949344 19779 raft_consensus.cc:1079] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Signalling peer 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:57:00.950865 19648 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:35206
I20250831 13:57:00.951342 19648 raft_consensus.cc:491] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:00.951625 19648 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:00.955869 19648 raft_consensus.cc:513] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:00.957597 19648 leader_election.cc:290] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 3 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:57:00.958537 19574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:57:00.959131 19574 raft_consensus.cc:3053] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:00.959439 19574 raft_consensus.cc:738] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:00.959394 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:00.960143 19499 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:00.960428 19574 consensus_queue.cc:260] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:00.961710 19574 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:00.967212 19499 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 3.
I20250831 13:57:00.968365 19615 leader_election.cc:304] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:57:00.968386 19574 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 3.
I20250831 13:57:00.969126 19697 raft_consensus.cc:2802] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:00.970479 19697 raft_consensus.cc:695] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:57:00.971336 19697 consensus_queue.cc:237] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:00.977630 19396 catalog_manager.cc:5582] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 2 to 3, leader changed from c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130) to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:01.029104 19704 raft_consensus.cc:991] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b: : Instructing follower 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:57:01.029492 19704 raft_consensus.cc:1079] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Signalling peer 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:57:01.030741 19648 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:35220
I20250831 13:57:01.031208 19648 raft_consensus.cc:491] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:01.031499 19648 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:01.035797 19648 raft_consensus.cc:513] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:01.037174 19648 leader_election.cc:290] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 3 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:57:01.037953 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:01.038188 19574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:57:01.038405 19499 raft_consensus.cc:3053] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:01.038661 19499 raft_consensus.cc:738] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:57:01.038662 19574 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:01.039307 19499 consensus_queue.cc:260] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:01.040143 19499 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:01.042992 19574 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 3.
I20250831 13:57:01.044008 19615 leader_election.cc:304] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:57:01.044298 19499 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 3.
I20250831 13:57:01.044900 19697 raft_consensus.cc:2802] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:01.045342 19697 raft_consensus.cc:695] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:57:01.046059 19697 consensus_queue.cc:237] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:01.052534 19396 catalog_manager.cc:5582] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 2 to 3, leader changed from 6efac5f24fe647729a762108877cab9b (127.18.233.129) to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:01.392966 19499 raft_consensus.cc:1273] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:01.394145 19697 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:01.402315 19574 raft_consensus.cc:1273] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:01.403820 19697 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:01.430058 19574 raft_consensus.cc:1273] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:01.431205 19697 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:01.439075 19499 raft_consensus.cc:1273] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:01.440544 19794 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:01.661851 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:01.667083 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:01.667464 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:02.668337 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:02.673627 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:02.674163 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:03.675110 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:03.679956 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:03.680367 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:04.681489 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:04.686527 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:04.686868 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:05.687903 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:05.692375 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:05.692682 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:06.693499 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:06.697955 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:06.698288 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:07.699191 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:07.703629 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:07.703938 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:08.704711 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:08.710139 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:08.710644 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:09.711514 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:09.716176 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:09.716564 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:10.742231 19366 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20250831 13:57:10.756258 19500 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
mode: GRACEFUL
new_leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:33784
I20250831 13:57:10.756850 19500 raft_consensus.cc:604] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Received request to transfer leadership to TS c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:57:10.761555 19500 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
mode: GRACEFUL
new_leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:33784
I20250831 13:57:10.761991 19500 raft_consensus.cc:604] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 1 LEADER]: Received request to transfer leadership to TS c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:57:10.766017 19648 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
mode: GRACEFUL
new_leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:35222
I20250831 13:57:10.766444 19648 raft_consensus.cc:604] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Received request to transfer leadership to TS c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:57:10.771296 19500 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ff828664880747f0831d670d6befb592"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
mode: GRACEFUL
new_leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:33784
I20250831 13:57:10.771745 19500 raft_consensus.cc:604] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Received request to transfer leadership to TS c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:57:10.778893 19648 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
mode: GRACEFUL
new_leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:35222
I20250831 13:57:10.779199 19805 raft_consensus.cc:991] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b: : Instructing follower c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:10.779376 19648 raft_consensus.cc:604] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Received request to transfer leadership to TS c6bcc1bb41fa4645b81d2c10907edfa0
I20250831 13:57:10.779788 19802 raft_consensus.cc:1079] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 1 LEADER]: Signalling peer c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:10.781139 19574 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:36202
I20250831 13:57:10.781520 19574 raft_consensus.cc:491] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:10.781819 19574 raft_consensus.cc:3058] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:10.799247 19574 raft_consensus.cc:513] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:10.800765 19574 leader_election.cc:290] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 2 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:57:10.801616 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:10.801978 19648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74df8cf8627043ca9caa1c1b7c441fb3" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:57:10.802237 19500 raft_consensus.cc:3053] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:10.802603 19648 raft_consensus.cc:3058] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:10.802596 19500 raft_consensus.cc:738] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:57:10.803519 19500 consensus_queue.cc:260] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:10.804457 19500 raft_consensus.cc:3058] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:10.851769 19805 raft_consensus.cc:991] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b: : Instructing follower c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:10.852260 19802 raft_consensus.cc:1079] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Signalling peer c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:10.853498 19574 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:36202
I20250831 13:57:10.853885 19574 raft_consensus.cc:491] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:10.854202 19574 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:10.899015 19648 raft_consensus.cc:2466] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 2.
I20250831 13:57:10.899323 19574 raft_consensus.cc:513] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:57:10.899015 19500 raft_consensus.cc:2466] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 2.
I20250831 13:57:10.900529 19540 leader_election.cc:304] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:57:10.901641 19701 raft_consensus.cc:2802] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:10.901855 19574 leader_election.cc:290] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 3 election: Requested vote from peers 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591), 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233)
I20250831 13:57:10.903669 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:10.904379 19499 raft_consensus.cc:3053] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:10.904737 19499 raft_consensus.cc:738] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:57:10.904788 19648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cdb3b7c54a244d82b47d49bbbd6f5f08" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:57:10.905480 19648 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:10.906068 19499 consensus_queue.cc:260] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:57:10.906524 19701 raft_consensus.cc:695] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:10.907437 19499 raft_consensus.cc:3058] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:10.907439 19701 consensus_queue.cc:237] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:10.915004 19397 catalog_manager.cc:5582] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 1 to 2, leader changed from 6efac5f24fe647729a762108877cab9b (127.18.233.129) to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:11.005362 19499 raft_consensus.cc:2466] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 3.
I20250831 13:57:11.005390 19648 raft_consensus.cc:2466] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 3.
I20250831 13:57:11.006661 19541 leader_election.cc:304] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:57:11.007616 19701 raft_consensus.cc:2802] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:11.007993 19701 raft_consensus.cc:695] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:11.008781 19701 consensus_queue.cc:237] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } }
I20250831 13:57:11.011736 19805 raft_consensus.cc:991] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b: : Instructing follower c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:11.012231 19802 raft_consensus.cc:1079] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Signalling peer c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:11.014146 19574 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ff828664880747f0831d670d6befb592"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:36202
I20250831 13:57:11.014606 19574 raft_consensus.cc:491] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:11.014955 19574 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:11.016517 19397 catalog_manager.cc:5582] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 2 to 3, leader changed from 6efac5f24fe647729a762108877cab9b (127.18.233.129) to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 3 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:11.108811 19574 raft_consensus.cc:513] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.110359 19574 leader_election.cc:290] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 3 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:57:11.111176 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:11.111723 19499 raft_consensus.cc:3053] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:11.111603 19648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:57:11.112003 19499 raft_consensus.cc:738] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:57:11.112260 19648 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:11.112676 19499 consensus_queue.cc:260] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.113616 19499 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:11.145501 19814 raft_consensus.cc:991] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626: : Instructing follower c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:11.146013 19796 raft_consensus.cc:1079] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Signalling peer c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:11.147244 19574 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:36196
I20250831 13:57:11.147667 19574 raft_consensus.cc:491] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:11.147898 19574 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:11.158262 19815 raft_consensus.cc:991] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626: : Instructing follower c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:11.158771 19796 raft_consensus.cc:1079] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Signalling peer c6bcc1bb41fa4645b81d2c10907edfa0 to start an election
I20250831 13:57:11.160164 19575 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
from {username='slave'} at 127.0.0.1:36196
I20250831 13:57:11.160835 19575 raft_consensus.cc:491] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:11.161175 19575 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:11.209249 19499 raft_consensus.cc:2466] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 3.
I20250831 13:57:11.209311 19648 raft_consensus.cc:2466] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 3.
I20250831 13:57:11.210719 19540 leader_election.cc:304] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:57:11.211297 19574 raft_consensus.cc:513] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.212149 19701 raft_consensus.cc:2802] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:11.212646 19701 raft_consensus.cc:695] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:11.213507 19701 consensus_queue.cc:237] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.214000 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:11.214579 19499 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:11.215085 19648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:57:11.215564 19648 raft_consensus.cc:3053] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:57:11.215816 19648 raft_consensus.cc:738] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:57:11.213897 19574 leader_election.cc:290] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 4 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:57:11.216506 19648 consensus_queue.cc:260] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.217578 19648 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:11.221381 19397 catalog_manager.cc:5582] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 2 to 3, leader changed from 6efac5f24fe647729a762108877cab9b (127.18.233.129) to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 3 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:11.312758 19499 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 4.
I20250831 13:57:11.312793 19575 raft_consensus.cc:513] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.312784 19648 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 4.
I20250831 13:57:11.314153 19541 leader_election.cc:304] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efac5f24fe647729a762108877cab9b, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:57:11.315183 19701 raft_consensus.cc:2802] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:57:11.315672 19701 raft_consensus.cc:695] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:11.315973 19575 leader_election.cc:290] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 4 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:57:11.316968 19648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:57:11.316970 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:11.317302 19701 consensus_queue.cc:237] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.317652 19648 raft_consensus.cc:3053] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:57:11.317708 19499 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:11.318173 19648 raft_consensus.cc:738] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:57:11.319729 19648 consensus_queue.cc:260] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.320771 19648 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:11.323716 19397 catalog_manager.cc:5582] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 3 to 4, leader changed from 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131) to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 4 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:11.377188 19500 raft_consensus.cc:1273] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:11.378358 19701 consensus_queue.cc:1035] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:11.389307 19650 raft_consensus.cc:1273] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:11.390650 19806 consensus_queue.cc:1035] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:57:11.410504 19499 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 4.
I20250831 13:57:11.410504 19648 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bcc1bb41fa4645b81d2c10907edfa0 in term 4.
I20250831 13:57:11.411733 19540 leader_election.cc:304] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, c6bcc1bb41fa4645b81d2c10907edfa0; no voters:
I20250831 13:57:11.412681 19806 raft_consensus.cc:2802] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:57:11.413161 19806 raft_consensus.cc:695] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Becoming Leader. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:11.413856 19806 consensus_queue.cc:237] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:11.420540 19396 catalog_manager.cc:5582] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 reported cstate change: term changed from 3 to 4, leader changed from 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131) to c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130). New cstate: current_term: 4 leader_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:11.548938 19648 raft_consensus.cc:1273] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:11.550073 19701 consensus_queue.cc:1035] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:11.557484 19499 raft_consensus.cc:1273] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:11.559288 19701 consensus_queue.cc:1035] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:11.701505 19499 raft_consensus.cc:1273] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:11.702558 19701 consensus_queue.cc:1035] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:11.709656 19648 raft_consensus.cc:1273] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:11.711309 19806 consensus_queue.cc:1035] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:11.793377 19648 raft_consensus.cc:1273] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:11.794464 19806 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:11.802134 19499 raft_consensus.cc:1273] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:11.803550 19701 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:11.848004 19650 raft_consensus.cc:1273] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:11.849282 19830 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:11.857867 19499 raft_consensus.cc:1273] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Refusing update from remote peer c6bcc1bb41fa4645b81d2c10907edfa0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:11.859978 19830 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:13.811450 19366 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20250831 13:57:13.812081 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:13.818552 19575 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
mode: GRACEFUL
new_leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:36218
I20250831 13:57:13.819020 19575 raft_consensus.cc:604] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Received request to transfer leadership to TS 5ce6fc4a86d04c5388b840b1b18c3626
I20250831 13:57:13.821056 19575 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
mode: GRACEFUL
new_leader_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:36218
I20250831 13:57:13.821468 19575 raft_consensus.cc:604] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Received request to transfer leadership to TS 6efac5f24fe647729a762108877cab9b
I20250831 13:57:13.823407 19575 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
mode: GRACEFUL
new_leader_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:36218
I20250831 13:57:13.823800 19575 raft_consensus.cc:604] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Received request to transfer leadership to TS 6efac5f24fe647729a762108877cab9b
I20250831 13:57:13.825779 19575 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f934026fa41446ac9c7ad9aa51e2fa3d"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
mode: GRACEFUL
new_leader_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:36218
I20250831 13:57:13.826184 19575 raft_consensus.cc:604] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Received request to transfer leadership to TS 6efac5f24fe647729a762108877cab9b
I20250831 13:57:13.828423 19575 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ff828664880747f0831d670d6befb592"
dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
mode: GRACEFUL
new_leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:36218
I20250831 13:57:13.828831 19575 raft_consensus.cc:604] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 LEADER]: Received request to transfer leadership to TS 5ce6fc4a86d04c5388b840b1b18c3626
I20250831 13:57:13.829800 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250831 13:57:13.830196 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:13.848676 19833 raft_consensus.cc:991] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0: : Instructing follower 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:57:13.849090 19830 raft_consensus.cc:1079] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Signalling peer 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:57:13.850464 19650 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:35206
I20250831 13:57:13.850893 19650 raft_consensus.cc:491] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:13.851135 19650 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:13.855654 19650 raft_consensus.cc:513] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:13.857537 19650 leader_election.cc:290] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 3 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:57:13.858067 19500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:13.858393 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c5f9bcfbd354d6b8fc27a930266a723" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:57:13.858603 19500 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:13.858938 19575 raft_consensus.cc:3053] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:13.859193 19575 raft_consensus.cc:738] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:13.859917 19575 consensus_queue.cc:260] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:13.860947 19575 raft_consensus.cc:3058] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:13.862857 19500 raft_consensus.cc:2466] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 3.
I20250831 13:57:13.863827 19615 leader_election.cc:304] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:57:13.864501 19796 raft_consensus.cc:2802] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:13.865360 19796 raft_consensus.cc:695] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:57:13.865584 19575 raft_consensus.cc:2466] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 3.
I20250831 13:57:13.866168 19796 consensus_queue.cc:237] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:13.871706 19397 catalog_manager.cc:5582] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 2 to 3, leader changed from c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130) to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:14.057221 19833 raft_consensus.cc:991] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0: : Instructing follower 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:57:14.057613 19830 raft_consensus.cc:1079] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Signalling peer 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:57:14.058825 19499 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:59366
I20250831 13:57:14.059228 19499 raft_consensus.cc:491] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:14.059474 19499 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:14.063206 19499 raft_consensus.cc:513] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.064728 19499 leader_election.cc:290] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 5 election: Requested vote from peers c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:57:14.065508 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:57:14.065606 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bab0ce4153b84bf59b45f7788ff29c5f" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:57:14.066093 19575 raft_consensus.cc:3053] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:57:14.066190 19650 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:14.066390 19575 raft_consensus.cc:738] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:14.067073 19575 consensus_queue.cc:260] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.067857 19575 raft_consensus.cc:3058] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:14.070644 19650 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 5.
I20250831 13:57:14.071638 19465 leader_election.cc:304] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:57:14.072034 19575 raft_consensus.cc:2466] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 5.
I20250831 13:57:14.073119 19840 raft_consensus.cc:2802] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:57:14.074110 19840 raft_consensus.cc:695] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 5 LEADER]: Becoming Leader. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:57:14.074786 19840 consensus_queue.cc:237] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.080499 19397 catalog_manager.cc:5582] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b reported cstate change: term changed from 4 to 5, leader changed from c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130) to 6efac5f24fe647729a762108877cab9b (127.18.233.129). New cstate: current_term: 5 leader_uuid: "6efac5f24fe647729a762108877cab9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:14.095629 19833 raft_consensus.cc:991] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0: : Instructing follower 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:57:14.095981 19830 raft_consensus.cc:1079] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 LEADER]: Signalling peer 5ce6fc4a86d04c5388b840b1b18c3626 to start an election
I20250831 13:57:14.097141 19650 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ff828664880747f0831d670d6befb592"
dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
from {username='slave'} at 127.0.0.1:35206
I20250831 13:57:14.097589 19650 raft_consensus.cc:491] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:14.097858 19650 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:14.101641 19650 raft_consensus.cc:513] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.102877 19650 leader_election.cc:290] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 4 election: Requested vote from peers 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233), c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797)
I20250831 13:57:14.103560 19499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6efac5f24fe647729a762108877cab9b"
I20250831 13:57:14.103842 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff828664880747f0831d670d6befb592" candidate_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:57:14.103987 19499 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:14.104277 19575 raft_consensus.cc:3053] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:57:14.104496 19575 raft_consensus.cc:738] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:14.105046 19575 consensus_queue.cc:260] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.105798 19575 raft_consensus.cc:3058] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:14.107869 19499 raft_consensus.cc:2466] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 4.
I20250831 13:57:14.108636 19615 leader_election.cc:304] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:57:14.109175 19796 raft_consensus.cc:2802] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:57:14.109491 19796 raft_consensus.cc:695] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 LEADER]: Becoming Leader. State: Replica: 5ce6fc4a86d04c5388b840b1b18c3626, State: Running, Role: LEADER
I20250831 13:57:14.109486 19575 raft_consensus.cc:2466] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ce6fc4a86d04c5388b840b1b18c3626 in term 4.
I20250831 13:57:14.110188 19796 consensus_queue.cc:237] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.116560 19397 catalog_manager.cc:5582] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 reported cstate change: term changed from 3 to 4, leader changed from c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130) to 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131). New cstate: current_term: 4 leader_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:14.162541 19833 raft_consensus.cc:991] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0: : Instructing follower 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:57:14.162957 19830 raft_consensus.cc:1079] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Signalling peer 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:57:14.164084 19499 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f934026fa41446ac9c7ad9aa51e2fa3d"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:59366
I20250831 13:57:14.164448 19499 raft_consensus.cc:491] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:14.164657 19499 raft_consensus.cc:3058] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:14.168735 19499 raft_consensus.cc:513] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.169993 19499 leader_election.cc:290] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 2 election: Requested vote from peers c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:57:14.170701 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f934026fa41446ac9c7ad9aa51e2fa3d" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:57:14.170939 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f934026fa41446ac9c7ad9aa51e2fa3d" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:57:14.171175 19575 raft_consensus.cc:3053] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:14.171495 19575 raft_consensus.cc:738] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:14.172248 19650 raft_consensus.cc:3058] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:14.172267 19575 consensus_queue.cc:260] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.173473 19575 raft_consensus.cc:3058] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:14.178289 19650 raft_consensus.cc:2466] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 2.
I20250831 13:57:14.190191 19465 leader_election.cc:304] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:57:14.192003 19840 raft_consensus.cc:2802] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:14.192920 19840 raft_consensus.cc:695] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Becoming Leader. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:57:14.194754 19840 consensus_queue.cc:237] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.197489 19575 raft_consensus.cc:2466] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 2.
I20250831 13:57:14.205868 19397 catalog_manager.cc:5582] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b reported cstate change: term changed from 1 to 2, leader changed from c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130) to 6efac5f24fe647729a762108877cab9b (127.18.233.129). New cstate: current_term: 2 leader_uuid: "6efac5f24fe647729a762108877cab9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:14.325486 19833 raft_consensus.cc:991] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0: : Instructing follower 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:57:14.325994 19830 raft_consensus.cc:1079] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Signalling peer 6efac5f24fe647729a762108877cab9b to start an election
I20250831 13:57:14.327368 19499 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb"
dest_uuid: "6efac5f24fe647729a762108877cab9b"
from {username='slave'} at 127.0.0.1:59366
I20250831 13:57:14.327823 19499 raft_consensus.cc:491] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:14.328104 19499 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:14.332535 19499 raft_consensus.cc:513] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.333755 19499 leader_election.cc:290] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 5 election: Requested vote from peers c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130:35797), 5ce6fc4a86d04c5388b840b1b18c3626 (127.18.233.131:41591)
I20250831 13:57:14.335048 19575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0"
I20250831 13:57:14.335211 19650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c9cea365d40299758f1dbcfa1d5bb" candidate_uuid: "6efac5f24fe647729a762108877cab9b" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "5ce6fc4a86d04c5388b840b1b18c3626"
I20250831 13:57:14.335695 19575 raft_consensus.cc:3053] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:57:14.335824 19650 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:14.335971 19575 raft_consensus.cc:738] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c6bcc1bb41fa4645b81d2c10907edfa0, State: Running, Role: LEADER
I20250831 13:57:14.336627 19575 consensus_queue.cc:260] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.337679 19575 raft_consensus.cc:3058] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:14.341764 19650 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 5.
I20250831 13:57:14.342572 19575 raft_consensus.cc:2466] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efac5f24fe647729a762108877cab9b in term 5.
I20250831 13:57:14.342712 19465 leader_election.cc:304] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ce6fc4a86d04c5388b840b1b18c3626, 6efac5f24fe647729a762108877cab9b; no voters:
I20250831 13:57:14.343494 19840 raft_consensus.cc:2802] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:57:14.343844 19840 raft_consensus.cc:695] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 5 LEADER]: Becoming Leader. State: Replica: 6efac5f24fe647729a762108877cab9b, State: Running, Role: LEADER
I20250831 13:57:14.344450 19840 consensus_queue.cc:237] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } }
I20250831 13:57:14.350782 19575 raft_consensus.cc:1273] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:14.351435 19397 catalog_manager.cc:5582] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b reported cstate change: term changed from 4 to 5, leader changed from c6bcc1bb41fa4645b81d2c10907edfa0 (127.18.233.130) to 6efac5f24fe647729a762108877cab9b (127.18.233.129). New cstate: current_term: 5 leader_uuid: "6efac5f24fe647729a762108877cab9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:14.352098 19796 consensus_queue.cc:1035] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:14.361681 19499 raft_consensus.cc:1273] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:14.363749 19834 consensus_queue.cc:1035] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:57:14.539036 19575 raft_consensus.cc:1273] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 5 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 13:57:14.540071 19840 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:57:14.548496 19650 raft_consensus.cc:1273] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 5 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 13:57:14.549767 19840 consensus_queue.cc:1035] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:57:14.611459 19575 raft_consensus.cc:1273] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:14.612675 19840 consensus_queue.cc:1035] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:14.612838 19500 raft_consensus.cc:1273] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:14.614020 19834 consensus_queue.cc:1035] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efac5f24fe647729a762108877cab9b" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 38233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:14.624680 19650 raft_consensus.cc:1273] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:14.625311 19575 raft_consensus.cc:1273] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Refusing update from remote peer 5ce6fc4a86d04c5388b840b1b18c3626: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:14.626971 19841 consensus_queue.cc:1035] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:57:14.628237 19834 consensus_queue.cc:1035] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:14.831447 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:14.837306 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:14.837679 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:14.852366 19650 raft_consensus.cc:1273] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 5 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 13:57:14.853472 19841 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ce6fc4a86d04c5388b840b1b18c3626" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 41591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:57:14.862246 19574 raft_consensus.cc:1273] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 5 FOLLOWER]: Refusing update from remote peer 6efac5f24fe647729a762108877cab9b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 13:57:14.864190 19858 consensus_queue.cc:1035] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bcc1bb41fa4645b81d2c10907edfa0" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:57:15.838845 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:15.843411 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:15.843814 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:16.844776 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:16.850360 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:16.850708 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:17.851558 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:17.856042 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:17.856367 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:18.857321 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:18.861794 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:18.862178 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:19.863116 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:19.867967 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:19.868357 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:20.869314 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:20.875248 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:20.875630 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:21.876628 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:21.880999 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:21.881402 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:22.882579 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:22.887423 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:22.887816 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:23.913833 19366 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20250831 13:57:23.914379 19366 tablet_server.cc:178] TabletServer@127.18.233.129:0 shutting down...
W20250831 13:57:23.927258 19541 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250831 13:57:23.932045 19541 consensus_peers.cc:489] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 -> Peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Couldn't send request to peer 6efac5f24fe647729a762108877cab9b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:57:23.942130 19366 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 13:57:23.942818 19366 tablet_replica.cc:331] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.943869 19366 raft_consensus.cc:2241] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:23.944471 19366 raft_consensus.cc:2270] T 74df8cf8627043ca9caa1c1b7c441fb3 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:23.946616 19366 tablet_replica.cc:331] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.947139 19366 raft_consensus.cc:2241] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 5 LEADER]: Raft consensus shutting down.
I20250831 13:57:23.947722 19366 raft_consensus.cc:2270] T ac5c9cea365d40299758f1dbcfa1d5bb P 6efac5f24fe647729a762108877cab9b [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:23.949153 19366 tablet_replica.cc:331] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.949579 19366 raft_consensus.cc:2241] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:23.949924 19366 raft_consensus.cc:2270] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:23.951426 19366 tablet_replica.cc:331] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.951856 19366 raft_consensus.cc:2241] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 5 LEADER]: Raft consensus shutting down.
I20250831 13:57:23.952481 19366 raft_consensus.cc:2270] T bab0ce4153b84bf59b45f7788ff29c5f P 6efac5f24fe647729a762108877cab9b [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:23.954066 19366 tablet_replica.cc:331] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.954586 19366 raft_consensus.cc:2241] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:23.954942 19366 raft_consensus.cc:2270] T 1c5f9bcfbd354d6b8fc27a930266a723 P 6efac5f24fe647729a762108877cab9b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:23.956171 19366 tablet_replica.cc:331] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.956636 19366 raft_consensus.cc:2241] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:23.956950 19366 raft_consensus.cc:2270] T 1bfadbd653f845489dcce6b6b819e776 P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:23.958359 19366 tablet_replica.cc:331] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.958825 19366 raft_consensus.cc:2241] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:23.959149 19366 raft_consensus.cc:2270] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 6efac5f24fe647729a762108877cab9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:23.960496 19366 tablet_replica.cc:331] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.960870 19366 raft_consensus.cc:2241] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 2 LEADER]: Raft consensus shutting down.
I20250831 13:57:23.961441 19366 raft_consensus.cc:2270] T f934026fa41446ac9c7ad9aa51e2fa3d P 6efac5f24fe647729a762108877cab9b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:23.963155 19366 tablet_replica.cc:331] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b: stopping tablet replica
I20250831 13:57:23.963593 19366 raft_consensus.cc:2241] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:23.964013 19366 raft_consensus.cc:2270] T ff828664880747f0831d670d6befb592 P 6efac5f24fe647729a762108877cab9b [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.043704 19366 tablet_server.cc:195] TabletServer@127.18.233.129:0 shutdown complete.
I20250831 13:57:24.060354 19366 tablet_server.cc:178] TabletServer@127.18.233.130:0 shutting down...
I20250831 13:57:24.085224 19366 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 13:57:24.085820 19366 tablet_replica.cc:331] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.086452 19366 raft_consensus.cc:2241] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 LEADER]: Raft consensus shutting down.
I20250831 13:57:24.087267 19366 raft_consensus.cc:2270] T 74df8cf8627043ca9caa1c1b7c441fb3 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.089154 19366 tablet_replica.cc:331] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.089677 19366 raft_consensus.cc:2241] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 LEADER]: Raft consensus shutting down.
I20250831 13:57:24.090490 19366 raft_consensus.cc:2270] T 1bfa452f0b4b40b09a0e8adee0a0afbe P c6bcc1bb41fa4645b81d2c10907edfa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.092572 19366 tablet_replica.cc:331] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.092990 19366 raft_consensus.cc:2241] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.093349 19366 raft_consensus.cc:2270] T f934026fa41446ac9c7ad9aa51e2fa3d P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.094765 19366 tablet_replica.cc:331] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.095194 19366 raft_consensus.cc:2241] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.095489 19366 raft_consensus.cc:2270] T ff828664880747f0831d670d6befb592 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.096679 19366 tablet_replica.cc:331] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.097064 19366 raft_consensus.cc:2241] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.097398 19366 raft_consensus.cc:2270] T bab0ce4153b84bf59b45f7788ff29c5f P c6bcc1bb41fa4645b81d2c10907edfa0 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.098634 19366 tablet_replica.cc:331] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.099112 19366 raft_consensus.cc:2241] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 LEADER]: Raft consensus shutting down.
I20250831 13:57:24.099764 19366 raft_consensus.cc:2270] T cdb3b7c54a244d82b47d49bbbd6f5f08 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.101166 19366 tablet_replica.cc:331] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.101545 19366 raft_consensus.cc:2241] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.101846 19366 raft_consensus.cc:2270] T ac5c9cea365d40299758f1dbcfa1d5bb P c6bcc1bb41fa4645b81d2c10907edfa0 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.103240 19366 tablet_replica.cc:331] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.103611 19366 raft_consensus.cc:2241] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.103943 19366 raft_consensus.cc:2270] T 1bfadbd653f845489dcce6b6b819e776 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.105320 19366 tablet_replica.cc:331] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0: stopping tablet replica
I20250831 13:57:24.105722 19366 raft_consensus.cc:2241] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.106091 19366 raft_consensus.cc:2270] T 1c5f9bcfbd354d6b8fc27a930266a723 P c6bcc1bb41fa4645b81d2c10907edfa0 [term 3 FOLLOWER]: Raft consensus is shut down!
W20250831 13:57:24.126811 19615 consensus_peers.cc:489] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 -> Peer 6efac5f24fe647729a762108877cab9b (127.18.233.129:38233): Couldn't send request to peer 6efac5f24fe647729a762108877cab9b. Status: Network error: Client connection negotiation failed: client connection to 127.18.233.129:38233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250831 13:57:24.131574 19366 tablet_server.cc:195] TabletServer@127.18.233.130:0 shutdown complete.
I20250831 13:57:24.148133 19366 tablet_server.cc:178] TabletServer@127.18.233.131:0 shutting down...
I20250831 13:57:24.171861 19366 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 13:57:24.172470 19366 tablet_replica.cc:331] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.173012 19366 raft_consensus.cc:2241] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.173450 19366 raft_consensus.cc:2270] T 1bfa452f0b4b40b09a0e8adee0a0afbe P 5ce6fc4a86d04c5388b840b1b18c3626 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.175101 19366 tablet_replica.cc:331] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.175522 19366 raft_consensus.cc:2241] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 LEADER]: Raft consensus shutting down.
I20250831 13:57:24.176216 19366 raft_consensus.cc:2270] T 1bfadbd653f845489dcce6b6b819e776 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.177616 19366 tablet_replica.cc:331] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.178045 19366 raft_consensus.cc:2241] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 LEADER]: Raft consensus shutting down.
I20250831 13:57:24.178689 19366 raft_consensus.cc:2270] T 1c5f9bcfbd354d6b8fc27a930266a723 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.180253 19366 tablet_replica.cc:331] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.180718 19366 raft_consensus.cc:2241] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.181120 19366 raft_consensus.cc:2270] T cdb3b7c54a244d82b47d49bbbd6f5f08 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.182600 19366 tablet_replica.cc:331] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.183107 19366 raft_consensus.cc:2241] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.183476 19366 raft_consensus.cc:2270] T ac5c9cea365d40299758f1dbcfa1d5bb P 5ce6fc4a86d04c5388b840b1b18c3626 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.184840 19366 tablet_replica.cc:331] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.185309 19366 raft_consensus.cc:2241] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.185675 19366 raft_consensus.cc:2270] T bab0ce4153b84bf59b45f7788ff29c5f P 5ce6fc4a86d04c5388b840b1b18c3626 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.187137 19366 tablet_replica.cc:331] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.187548 19366 raft_consensus.cc:2241] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.187865 19366 raft_consensus.cc:2270] T 74df8cf8627043ca9caa1c1b7c441fb3 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.189226 19366 tablet_replica.cc:331] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.189682 19366 raft_consensus.cc:2241] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:24.190006 19366 raft_consensus.cc:2270] T f934026fa41446ac9c7ad9aa51e2fa3d P 5ce6fc4a86d04c5388b840b1b18c3626 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.191365 19366 tablet_replica.cc:331] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626: stopping tablet replica
I20250831 13:57:24.191762 19366 raft_consensus.cc:2241] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 LEADER]: Raft consensus shutting down.
I20250831 13:57:24.192409 19366 raft_consensus.cc:2270] T ff828664880747f0831d670d6befb592 P 5ce6fc4a86d04c5388b840b1b18c3626 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.230103 19366 tablet_server.cc:195] TabletServer@127.18.233.131:0 shutdown complete.
I20250831 13:57:24.250787 19366 master.cc:561] Master@127.18.233.190:40083 shutting down...
I20250831 13:57:24.272202 19366 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 1 LEADER]: Raft consensus shutting down.
I20250831 13:57:24.272969 19366 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:24.273321 19366 tablet_replica.cc:331] T 00000000000000000000000000000000 P 525c9ed220ce4bb79c08c08ca2199b07: stopping tablet replica
I20250831 13:57:24.292546 19366 master.cc:583] Master@127.18.233.190:40083 shutdown complete.
5ce6fc4a86d04c5388b840b1b18c3626 1
6efac5f24fe647729a762108877cab9b 4
c6bcc1bb41fa4645b81d2c10907edfa0 4
5ce6fc4a86d04c5388b840b1b18c3626 3
6efac5f24fe647729a762108877cab9b 3
c6bcc1bb41fa4645b81d2c10907edfa0 3
5ce6fc4a86d04c5388b840b1b18c3626 1
6efac5f24fe647729a762108877cab9b 0
c6bcc1bb41fa4645b81d2c10907edfa0 8
5ce6fc4a86d04c5388b840b1b18c3626 3
6efac5f24fe647729a762108877cab9b 3
c6bcc1bb41fa4645b81d2c10907edfa0 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (30213 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20250831 13:57:24.335693 19366 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.233.190:39183
I20250831 13:57:24.336761 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:57:24.342454 19872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:24.342623 19873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:24.343461 19875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:24.343739 19366 server_base.cc:1047] running on GCE node
I20250831 13:57:24.344660 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:57:24.344842 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:57:24.345012 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648644344996 us; error 0 us; skew 500 ppm
I20250831 13:57:24.345533 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:57:24.348335 19366 webserver.cc:480] Webserver started at http://127.18.233.190:43325/ using document root <none> and password file <none>
I20250831 13:57:24.348809 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:57:24.348965 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:57:24.349227 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:57:24.350391 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/master-0-root/instance:
uuid: "7f907f12a2e34b21a5596a485e246e78"
format_stamp: "Formatted at 2025-08-31 13:57:24 on dist-test-slave-sg9n"
I20250831 13:57:24.354610 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250831 13:57:24.357822 19880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.358520 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250831 13:57:24.358796 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/master-0-root
uuid: "7f907f12a2e34b21a5596a485e246e78"
format_stamp: "Formatted at 2025-08-31 13:57:24 on dist-test-slave-sg9n"
I20250831 13:57:24.359062 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:57:24.370497 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:57:24.371698 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:57:24.407976 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.190:39183
I20250831 13:57:24.408064 19931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.190:39183 every 8 connection(s)
I20250831 13:57:24.411715 19932 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.420750 19932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78: Bootstrap starting.
I20250831 13:57:24.425264 19932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:24.429169 19932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78: No bootstrap required, opened a new log
I20250831 13:57:24.431229 19932 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f907f12a2e34b21a5596a485e246e78" member_type: VOTER }
I20250831 13:57:24.431638 19932 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:24.431867 19932 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f907f12a2e34b21a5596a485e246e78, State: Initialized, Role: FOLLOWER
I20250831 13:57:24.432367 19932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f907f12a2e34b21a5596a485e246e78" member_type: VOTER }
I20250831 13:57:24.432796 19932 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250831 13:57:24.433010 19932 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250831 13:57:24.433279 19932 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:24.436941 19932 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f907f12a2e34b21a5596a485e246e78" member_type: VOTER }
I20250831 13:57:24.437424 19932 leader_election.cc:304] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f907f12a2e34b21a5596a485e246e78; no voters:
I20250831 13:57:24.438501 19932 leader_election.cc:290] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250831 13:57:24.438875 19935 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:24.440106 19935 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 1 LEADER]: Becoming Leader. State: Replica: 7f907f12a2e34b21a5596a485e246e78, State: Running, Role: LEADER
I20250831 13:57:24.440644 19935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f907f12a2e34b21a5596a485e246e78" member_type: VOTER }
I20250831 13:57:24.441255 19932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [sys.catalog]: configured and running, proceeding with master startup.
I20250831 13:57:24.447026 19936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f907f12a2e34b21a5596a485e246e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f907f12a2e34b21a5596a485e246e78" member_type: VOTER } }
I20250831 13:57:24.447197 19937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f907f12a2e34b21a5596a485e246e78. Latest consensus state: current_term: 1 leader_uuid: "7f907f12a2e34b21a5596a485e246e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f907f12a2e34b21a5596a485e246e78" member_type: VOTER } }
I20250831 13:57:24.447749 19937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [sys.catalog]: This master's current role is: LEADER
I20250831 13:57:24.447711 19936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [sys.catalog]: This master's current role is: LEADER
I20250831 13:57:24.449952 19945 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 13:57:24.457296 19945 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 13:57:24.458221 19366 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250831 13:57:24.465379 19945 catalog_manager.cc:1349] Generated new cluster ID: 4e65a6f165454d0bbd4dbfa4c5447d5f
I20250831 13:57:24.465590 19945 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 13:57:24.481161 19945 catalog_manager.cc:1372] Generated new certificate authority record
I20250831 13:57:24.482246 19945 catalog_manager.cc:1506] Loading token signing keys...
I20250831 13:57:24.503823 19945 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78: Generated new TSK 0
I20250831 13:57:24.504354 19945 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 13:57:24.524619 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:57:24.530279 19953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:24.531610 19954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:24.532670 19956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:24.533686 19366 server_base.cc:1047] running on GCE node
I20250831 13:57:24.534554 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:57:24.534761 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:57:24.534891 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648644534878 us; error 0 us; skew 500 ppm
I20250831 13:57:24.535343 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:57:24.537505 19366 webserver.cc:480] Webserver started at http://127.18.233.129:40109/ using document root <none> and password file <none>
I20250831 13:57:24.537951 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:57:24.538112 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:57:24.538317 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:57:24.539423 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-0-root/instance:
uuid: "998e50cfd9d94f17a96ecbf04b97496f"
format_stamp: "Formatted at 2025-08-31 13:57:24 on dist-test-slave-sg9n"
I20250831 13:57:24.543538 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250831 13:57:24.546481 19961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.547191 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250831 13:57:24.547437 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-0-root
uuid: "998e50cfd9d94f17a96ecbf04b97496f"
format_stamp: "Formatted at 2025-08-31 13:57:24 on dist-test-slave-sg9n"
I20250831 13:57:24.547663 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:57:24.564640 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:57:24.565783 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:57:24.567203 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:57:24.569145 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:57:24.569325 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.569558 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:57:24.569715 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.606948 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.129:37799
I20250831 13:57:24.607034 20023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.129:37799 every 8 connection(s)
I20250831 13:57:24.611495 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:57:24.618922 20028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:24.619524 20029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:24.621909 19366 server_base.cc:1047] running on GCE node
W20250831 13:57:24.622803 20032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:24.623699 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:57:24.623883 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:57:24.624007 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648644623996 us; error 0 us; skew 500 ppm
I20250831 13:57:24.624470 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:57:24.626961 19366 webserver.cc:480] Webserver started at http://127.18.233.130:46799/ using document root <none> and password file <none>
I20250831 13:57:24.627542 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:57:24.627753 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:57:24.628011 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:57:24.628422 20024 heartbeater.cc:344] Connected to a master server at 127.18.233.190:39183
I20250831 13:57:24.628731 20024 heartbeater.cc:461] Registering TS with master...
I20250831 13:57:24.629208 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-1-root/instance:
uuid: "0f6544748dcb4a1b88337dd4c55be644"
format_stamp: "Formatted at 2025-08-31 13:57:24 on dist-test-slave-sg9n"
I20250831 13:57:24.629395 20024 heartbeater.cc:507] Master 127.18.233.190:39183 requested a full tablet report, sending...
I20250831 13:57:24.631283 19897 ts_manager.cc:194] Registered new tserver with Master: 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799)
I20250831 13:57:24.632850 19897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57252
I20250831 13:57:24.633993 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250831 13:57:24.637316 20036 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.638001 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250831 13:57:24.638253 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-1-root
uuid: "0f6544748dcb4a1b88337dd4c55be644"
format_stamp: "Formatted at 2025-08-31 13:57:24 on dist-test-slave-sg9n"
I20250831 13:57:24.638504 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:57:24.681686 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:57:24.682920 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:57:24.684346 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:57:24.686509 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:57:24.686745 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250831 13:57:24.686991 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:57:24.687148 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.724129 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.130:37829
I20250831 13:57:24.724206 20098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.130:37829 every 8 connection(s)
I20250831 13:57:24.728648 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:57:24.735136 20102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:24.736711 20103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:24.740610 20105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:24.740761 20099 heartbeater.cc:344] Connected to a master server at 127.18.233.190:39183
I20250831 13:57:24.741168 20099 heartbeater.cc:461] Registering TS with master...
I20250831 13:57:24.741338 19366 server_base.cc:1047] running on GCE node
I20250831 13:57:24.741986 20099 heartbeater.cc:507] Master 127.18.233.190:39183 requested a full tablet report, sending...
I20250831 13:57:24.742208 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:57:24.742427 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:57:24.742578 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648644742561 us; error 0 us; skew 500 ppm
I20250831 13:57:24.743252 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:57:24.743877 19897 ts_manager.cc:194] Registered new tserver with Master: 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:24.745321 19897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57266
I20250831 13:57:24.745684 19366 webserver.cc:480] Webserver started at http://127.18.233.131:36925/ using document root <none> and password file <none>
I20250831 13:57:24.746222 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:57:24.746466 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:57:24.746766 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:57:24.747871 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-2-root/instance:
uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
format_stamp: "Formatted at 2025-08-31 13:57:24 on dist-test-slave-sg9n"
I20250831 13:57:24.752347 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250831 13:57:24.755275 20110 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.755936 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250831 13:57:24.756224 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-2-root
uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
format_stamp: "Formatted at 2025-08-31 13:57:24 on dist-test-slave-sg9n"
I20250831 13:57:24.756480 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1756648613964447-19366-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:57:24.770648 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:57:24.771773 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:57:24.773102 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:57:24.775213 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:57:24.775381 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.775637 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:57:24.775772 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:24.813017 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.131:35961
I20250831 13:57:24.813149 20172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.131:35961 every 8 connection(s)
I20250831 13:57:24.826201 20173 heartbeater.cc:344] Connected to a master server at 127.18.233.190:39183
I20250831 13:57:24.826550 20173 heartbeater.cc:461] Registering TS with master...
I20250831 13:57:24.827262 20173 heartbeater.cc:507] Master 127.18.233.190:39183 requested a full tablet report, sending...
I20250831 13:57:24.829000 19897 ts_manager.cc:194] Registered new tserver with Master: c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:24.829460 19366 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013268179s
I20250831 13:57:24.830296 19897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57272
I20250831 13:57:24.831899 19366 test_util.cc:276] Using random seed: -1569166710
I20250831 13:57:24.852661 19897 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57288:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250831 13:57:24.855455 19897 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 13:57:24.932531 19989 tablet_service.cc:1468] Processing CreateTablet for tablet 627caa15676449a4adc53cbb673ba506 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250831 13:57:24.933068 19988 tablet_service.cc:1468] Processing CreateTablet for tablet 01a91fa016854945a8cd940e2af5f258 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250831 13:57:24.933969 19989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627caa15676449a4adc53cbb673ba506. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.935317 19987 tablet_service.cc:1468] Processing CreateTablet for tablet 4443ba9be133499fb5f093495b583fab (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250831 13:57:24.936592 19987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4443ba9be133499fb5f093495b583fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.939819 19986 tablet_service.cc:1468] Processing CreateTablet for tablet 274f624b6e8d4c0ebbcf1bafbf5e5372 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250831 13:57:24.941048 19986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274f624b6e8d4c0ebbcf1bafbf5e5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.943318 20062 tablet_service.cc:1468] Processing CreateTablet for tablet 4443ba9be133499fb5f093495b583fab (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250831 13:57:24.944620 20062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4443ba9be133499fb5f093495b583fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.944535 20137 tablet_service.cc:1468] Processing CreateTablet for tablet 01a91fa016854945a8cd940e2af5f258 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250831 13:57:24.945991 20137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01a91fa016854945a8cd940e2af5f258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.948168 19984 tablet_service.cc:1468] Processing CreateTablet for tablet 096f3a0441b04af0b9352bf7ef8b42d3 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250831 13:57:24.948410 20135 tablet_service.cc:1468] Processing CreateTablet for tablet 274f624b6e8d4c0ebbcf1bafbf5e5372 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250831 13:57:24.951011 20063 tablet_service.cc:1468] Processing CreateTablet for tablet 01a91fa016854945a8cd940e2af5f258 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250831 13:57:24.951339 20138 tablet_service.cc:1468] Processing CreateTablet for tablet 627caa15676449a4adc53cbb673ba506 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250831 13:57:24.951745 19980 tablet_service.cc:1468] Processing CreateTablet for tablet 8124d84173894208b877ccf8285f5a82 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250831 13:57:24.952280 20063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01a91fa016854945a8cd940e2af5f258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.952661 20138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627caa15676449a4adc53cbb673ba506. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.953095 20131 tablet_service.cc:1468] Processing CreateTablet for tablet ae2e10d60e53469396554f7ff859aa48 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250831 13:57:24.953760 19983 tablet_service.cc:1468] Processing CreateTablet for tablet 35588c3345f746f791346a38eb2f2c31 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250831 13:57:24.954439 20131 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2e10d60e53469396554f7ff859aa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.955413 19982 tablet_service.cc:1468] Processing CreateTablet for tablet ae2e10d60e53469396554f7ff859aa48 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250831 13:57:24.955468 20133 tablet_service.cc:1468] Processing CreateTablet for tablet 096f3a0441b04af0b9352bf7ef8b42d3 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250831 13:57:24.955742 19985 tablet_service.cc:1468] Processing CreateTablet for tablet f0e0a1ba07754b09a1f73320114c37db (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250831 13:57:24.948766 19981 tablet_service.cc:1468] Processing CreateTablet for tablet 352060b4fee349d7b694f5bd230d1c0b (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250831 13:57:24.956816 20136 tablet_service.cc:1468] Processing CreateTablet for tablet 4443ba9be133499fb5f093495b583fab (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250831 13:57:24.957175 20064 tablet_service.cc:1468] Processing CreateTablet for tablet 627caa15676449a4adc53cbb673ba506 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250831 13:57:24.948822 20132 tablet_service.cc:1468] Processing CreateTablet for tablet 35588c3345f746f791346a38eb2f2c31 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250831 13:57:24.951261 20134 tablet_service.cc:1468] Processing CreateTablet for tablet f0e0a1ba07754b09a1f73320114c37db (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250831 13:57:24.958396 20064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 627caa15676449a4adc53cbb673ba506. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.959838 19988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01a91fa016854945a8cd940e2af5f258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.960525 19982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2e10d60e53469396554f7ff859aa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.960415 20061 tablet_service.cc:1468] Processing CreateTablet for tablet 274f624b6e8d4c0ebbcf1bafbf5e5372 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250831 13:57:24.961784 20061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274f624b6e8d4c0ebbcf1bafbf5e5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.961894 19981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352060b4fee349d7b694f5bd230d1c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.963047 19985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e0a1ba07754b09a1f73320114c37db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.965086 20060 tablet_service.cc:1468] Processing CreateTablet for tablet f0e0a1ba07754b09a1f73320114c37db (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250831 13:57:24.966369 20060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e0a1ba07754b09a1f73320114c37db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.966985 19983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35588c3345f746f791346a38eb2f2c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.967782 19984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 096f3a0441b04af0b9352bf7ef8b42d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.969321 19980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8124d84173894208b877ccf8285f5a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.969241 20130 tablet_service.cc:1468] Processing CreateTablet for tablet 352060b4fee349d7b694f5bd230d1c0b (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250831 13:57:24.972309 20129 tablet_service.cc:1468] Processing CreateTablet for tablet 8124d84173894208b877ccf8285f5a82 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250831 13:57:24.966986 20135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 274f624b6e8d4c0ebbcf1bafbf5e5372. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.975677 20129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8124d84173894208b877ccf8285f5a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.978997 20133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 096f3a0441b04af0b9352bf7ef8b42d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.981426 20130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352060b4fee349d7b694f5bd230d1c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.987031 20132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35588c3345f746f791346a38eb2f2c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.991052 20134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0e0a1ba07754b09a1f73320114c37db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:24.999152 20136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4443ba9be133499fb5f093495b583fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:25.012306 20058 tablet_service.cc:1468] Processing CreateTablet for tablet 35588c3345f746f791346a38eb2f2c31 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250831 13:57:25.012676 20057 tablet_service.cc:1468] Processing CreateTablet for tablet ae2e10d60e53469396554f7ff859aa48 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250831 13:57:25.013618 20058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35588c3345f746f791346a38eb2f2c31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:25.015578 20056 tablet_service.cc:1468] Processing CreateTablet for tablet 352060b4fee349d7b694f5bd230d1c0b (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250831 13:57:25.019109 20055 tablet_service.cc:1468] Processing CreateTablet for tablet 8124d84173894208b877ccf8285f5a82 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250831 13:57:25.021430 20059 tablet_service.cc:1468] Processing CreateTablet for tablet 096f3a0441b04af0b9352bf7ef8b42d3 (DEFAULT_TABLE table=test-workload [id=35368bb876de40e0bb8c829900ebfdd1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250831 13:57:25.022868 20057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae2e10d60e53469396554f7ff859aa48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:25.026988 20059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 096f3a0441b04af0b9352bf7ef8b42d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:25.030073 20055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8124d84173894208b877ccf8285f5a82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:25.030725 20056 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 352060b4fee349d7b694f5bd230d1c0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:25.068502 20188 tablet_bootstrap.cc:492] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.093720 20189 tablet_bootstrap.cc:492] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.108448 20188 tablet_bootstrap.cc:654] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.120993 20189 tablet_bootstrap.cc:654] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.135344 20190 tablet_bootstrap.cc:492] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.156994 20190 tablet_bootstrap.cc:654] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.161990 20188 tablet_bootstrap.cc:492] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.162341 20188 ts_tablet_manager.cc:1397] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.094s user 0.012s sys 0.015s
I20250831 13:57:25.162987 20190 tablet_bootstrap.cc:492] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.163379 20190 ts_tablet_manager.cc:1397] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.010s
I20250831 13:57:25.164129 20189 tablet_bootstrap.cc:492] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.164559 20189 ts_tablet_manager.cc:1397] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.005s
I20250831 13:57:25.164870 20188 raft_consensus.cc:357] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.165634 20188 raft_consensus.cc:383] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.165514 20190 raft_consensus.cc:357] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.165980 20188 raft_consensus.cc:738] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.166246 20190 raft_consensus.cc:383] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.166575 20190 raft_consensus.cc:738] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.166714 20188 consensus_queue.cc:260] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.167131 20189 raft_consensus.cc:357] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.167299 20190 consensus_queue.cc:260] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.167819 20189 raft_consensus.cc:383] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.168222 20189 raft_consensus.cc:738] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.169217 20189 consensus_queue.cc:260] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.173671 20099 heartbeater.cc:499] Master 127.18.233.190:39183 was elected leader, sending a full tablet report...
I20250831 13:57:25.174368 20190 ts_tablet_manager.cc:1428] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20250831 13:57:25.175333 20190 tablet_bootstrap.cc:492] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.176506 20189 ts_tablet_manager.cc:1428] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20250831 13:57:25.178015 20173 heartbeater.cc:499] Master 127.18.233.190:39183 was elected leader, sending a full tablet report...
I20250831 13:57:25.178797 20188 ts_tablet_manager.cc:1428] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.016s user 0.007s sys 0.010s
I20250831 13:57:25.181341 20189 tablet_bootstrap.cc:492] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.181736 20190 tablet_bootstrap.cc:654] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.183017 20024 heartbeater.cc:499] Master 127.18.233.190:39183 was elected leader, sending a full tablet report...
I20250831 13:57:25.185165 20188 tablet_bootstrap.cc:492] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.191426 20188 tablet_bootstrap.cc:654] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.196944 20189 tablet_bootstrap.cc:654] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.202975 20188 tablet_bootstrap.cc:492] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.203320 20190 tablet_bootstrap.cc:492] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.203444 20188 ts_tablet_manager.cc:1397] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.002s
I20250831 13:57:25.203874 20190 ts_tablet_manager.cc:1397] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.016s
I20250831 13:57:25.206327 20188 raft_consensus.cc:357] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.207159 20188 raft_consensus.cc:383] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.207561 20188 raft_consensus.cc:738] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.208246 20188 consensus_queue.cc:260] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.210422 20188 ts_tablet_manager.cc:1428] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:57:25.211355 20188 tablet_bootstrap.cc:492] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.207077 20190 raft_consensus.cc:357] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.214823 20190 raft_consensus.cc:383] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.215118 20190 raft_consensus.cc:738] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.215919 20189 tablet_bootstrap.cc:492] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.215847 20190 consensus_queue.cc:260] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.216404 20189 ts_tablet_manager.cc:1397] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.004s
I20250831 13:57:25.218107 20190 ts_tablet_manager.cc:1428] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250831 13:57:25.218953 20190 tablet_bootstrap.cc:492] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.219102 20189 raft_consensus.cc:357] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.219738 20189 raft_consensus.cc:383] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.219995 20189 raft_consensus.cc:738] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.220736 20189 consensus_queue.cc:260] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.223241 20189 ts_tablet_manager.cc:1428] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20250831 13:57:25.227170 20190 tablet_bootstrap.cc:654] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.224152 20189 tablet_bootstrap.cc:492] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.233119 20190 tablet_bootstrap.cc:492] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.233567 20190 ts_tablet_manager.cc:1397] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20250831 13:57:25.234874 20189 tablet_bootstrap.cc:654] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.235771 20188 tablet_bootstrap.cc:654] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.238330 20190 raft_consensus.cc:357] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.239333 20190 raft_consensus.cc:383] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.239698 20190 raft_consensus.cc:738] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.240715 20190 consensus_queue.cc:260] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.244810 20190 ts_tablet_manager.cc:1428] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20250831 13:57:25.245945 20190 tablet_bootstrap.cc:492] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.246807 20189 tablet_bootstrap.cc:492] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.247287 20189 ts_tablet_manager.cc:1397] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.015s
I20250831 13:57:25.251291 20189 raft_consensus.cc:357] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.252024 20189 raft_consensus.cc:383] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.252341 20189 raft_consensus.cc:738] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.253106 20189 consensus_queue.cc:260] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.254062 20190 tablet_bootstrap.cc:654] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.255532 20189 ts_tablet_manager.cc:1428] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250831 13:57:25.256467 20189 tablet_bootstrap.cc:492] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.259974 20188 tablet_bootstrap.cc:492] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.260628 20188 ts_tablet_manager.cc:1397] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.010s
I20250831 13:57:25.263325 20188 raft_consensus.cc:357] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.263998 20188 raft_consensus.cc:383] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.264299 20188 raft_consensus.cc:738] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.265071 20188 consensus_queue.cc:260] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.268051 20189 tablet_bootstrap.cc:654] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.268603 20188 ts_tablet_manager.cc:1428] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20250831 13:57:25.269536 20188 tablet_bootstrap.cc:492] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.269802 20190 tablet_bootstrap.cc:492] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.270303 20190 ts_tablet_manager.cc:1397] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.013s
I20250831 13:57:25.273808 20190 raft_consensus.cc:357] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.274819 20190 raft_consensus.cc:383] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.275303 20190 raft_consensus.cc:738] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.275645 20188 tablet_bootstrap.cc:654] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.276441 20190 consensus_queue.cc:260] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.280030 20190 ts_tablet_manager.cc:1428] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20250831 13:57:25.280929 20190 tablet_bootstrap.cc:492] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.281059 20189 tablet_bootstrap.cc:492] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.281692 20189 ts_tablet_manager.cc:1397] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.025s user 0.019s sys 0.000s
I20250831 13:57:25.285046 20189 raft_consensus.cc:357] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.285831 20189 raft_consensus.cc:383] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.286131 20189 raft_consensus.cc:738] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.286818 20189 consensus_queue.cc:260] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.287393 20190 tablet_bootstrap.cc:654] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.289121 20189 ts_tablet_manager.cc:1428] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 13:57:25.287410 20188 tablet_bootstrap.cc:492] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.290467 20188 ts_tablet_manager.cc:1397] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.010s
I20250831 13:57:25.291496 20189 tablet_bootstrap.cc:492] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.293533 20188 raft_consensus.cc:357] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.294041 20190 tablet_bootstrap.cc:492] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.294668 20190 ts_tablet_manager.cc:1397] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20250831 13:57:25.294211 20188 raft_consensus.cc:383] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.295640 20188 raft_consensus.cc:738] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.296339 20188 consensus_queue.cc:260] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.297506 20190 raft_consensus.cc:357] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.298219 20190 raft_consensus.cc:383] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.298269 20189 tablet_bootstrap.cc:654] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.298620 20190 raft_consensus.cc:738] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.298877 20188 ts_tablet_manager.cc:1428] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20250831 13:57:25.299811 20188 tablet_bootstrap.cc:492] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.299501 20190 consensus_queue.cc:260] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.301868 20190 ts_tablet_manager.cc:1428] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250831 13:57:25.302861 20190 tablet_bootstrap.cc:492] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.308903 20190 tablet_bootstrap.cc:654] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.309796 20189 tablet_bootstrap.cc:492] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.310225 20189 ts_tablet_manager.cc:1397] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20250831 13:57:25.313277 20194 raft_consensus.cc:491] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.313731 20194 raft_consensus.cc:513] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.318377 20194 leader_election.cc:290] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.319140 20188 tablet_bootstrap.cc:654] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.320577 20190 tablet_bootstrap.cc:492] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.321122 20190 ts_tablet_manager.cc:1397] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20250831 13:57:25.324627 20190 raft_consensus.cc:357] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.324821 20189 raft_consensus.cc:357] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.325567 20190 raft_consensus.cc:383] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.325889 20189 raft_consensus.cc:383] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.326146 20190 raft_consensus.cc:738] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.326351 20189 raft_consensus.cc:738] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.327419 20190 consensus_queue.cc:260] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.329758 20189 consensus_queue.cc:260] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.333353 20190 ts_tablet_manager.cc:1428] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250831 13:57:25.334687 20190 tablet_bootstrap.cc:492] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.336341 20189 ts_tablet_manager.cc:1428] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.026s user 0.005s sys 0.003s
I20250831 13:57:25.337253 20189 tablet_bootstrap.cc:492] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.342907 20190 tablet_bootstrap.cc:654] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.354776 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8124d84173894208b877ccf8285f5a82" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
W20250831 13:57:25.356377 19962 leader_election.cc:343] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.361115 20188 tablet_bootstrap.cc:492] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.361598 20188 ts_tablet_manager.cc:1397] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.062s user 0.017s sys 0.027s
I20250831 13:57:25.361768 20189 tablet_bootstrap.cc:654] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.364765 20195 raft_consensus.cc:491] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.365427 20195 raft_consensus.cc:513] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.366256 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8124d84173894208b877ccf8285f5a82" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644" is_pre_election: true
I20250831 13:57:25.367169 20074 raft_consensus.cc:2466] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 0.
I20250831 13:57:25.369032 19963 leader_election.cc:304] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters: c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:25.370056 20194 raft_consensus.cc:2802] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.370424 20194 raft_consensus.cc:491] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.370792 20194 raft_consensus.cc:3058] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.371086 20195 leader_election.cc:290] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.373342 20190 tablet_bootstrap.cc:492] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.373872 20190 ts_tablet_manager.cc:1397] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.017s
I20250831 13:57:25.364758 20188 raft_consensus.cc:357] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.374293 20188 raft_consensus.cc:383] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.374616 20188 raft_consensus.cc:738] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.375384 20188 consensus_queue.cc:260] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.377729 20194 raft_consensus.cc:513] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.380333 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8124d84173894208b877ccf8285f5a82" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:25.380853 20074 raft_consensus.cc:3058] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.381664 20190 raft_consensus.cc:357] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.388890 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8124d84173894208b877ccf8285f5a82" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.384531 20189 tablet_bootstrap.cc:492] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.385376 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.389786 20148 raft_consensus.cc:2466] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
W20250831 13:57:25.390745 19962 leader_election.cc:343] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.387367 20074 raft_consensus.cc:2466] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 1.
I20250831 13:57:25.387889 20194 leader_election.cc:290] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.390867 20037 leader_election.cc:304] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:25.391491 20189 ts_tablet_manager.cc:1397] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.008s
I20250831 13:57:25.391841 20195 raft_consensus.cc:2802] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.392030 19963 leader_election.cc:304] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters: c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:25.392261 20195 raft_consensus.cc:491] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.392601 20195 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.395416 20189 raft_consensus.cc:357] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.396585 20189 raft_consensus.cc:383] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.397011 20189 raft_consensus.cc:738] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.389083 20190 raft_consensus.cc:383] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.397650 20190 raft_consensus.cc:738] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.398634 20190 consensus_queue.cc:260] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.400838 20189 consensus_queue.cc:260] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.406112 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
W20250831 13:57:25.407827 20037 leader_election.cc:343] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.413374 20217 raft_consensus.cc:2802] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.413851 20217 raft_consensus.cc:695] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:25.414666 20217 consensus_queue.cc:237] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.417073 20218 raft_consensus.cc:491] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.404727 20195 raft_consensus.cc:513] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.417506 20218 raft_consensus.cc:513] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.420428 20190 ts_tablet_manager.cc:1428] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.046s user 0.005s sys 0.006s
I20250831 13:57:25.422415 20189 ts_tablet_manager.cc:1428] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.030s user 0.007s sys 0.002s
I20250831 13:57:25.422542 20195 leader_election.cc:290] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.423189 20188 ts_tablet_manager.cc:1428] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.061s user 0.029s sys 0.024s
I20250831 13:57:25.423544 20190 tablet_bootstrap.cc:492] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.424185 20188 tablet_bootstrap.cc:492] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.426414 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
I20250831 13:57:25.426738 20189 tablet_bootstrap.cc:492] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.430253 20190 tablet_bootstrap.cc:654] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.431011 20188 tablet_bootstrap.cc:654] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
W20250831 13:57:25.431838 20037 leader_election.cc:343] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.432360 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.432920 20148 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.433065 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
W20250831 13:57:25.434782 20037 leader_election.cc:343] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.441655 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.442509 20148 raft_consensus.cc:2466] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 1.
W20250831 13:57:25.443430 20037 leader_election.cc:343] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.445623 20037 leader_election.cc:304] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644; no voters: 998e50cfd9d94f17a96ecbf04b97496f, c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:25.443949 20218 leader_election.cc:290] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.445051 20196 raft_consensus.cc:491] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.449723 20195 raft_consensus.cc:491] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:57:25.451501 19944 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:57:25.465298 20195 raft_consensus.cc:513] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.446749 20196 raft_consensus.cc:513] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.468379 20195 leader_election.cc:290] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.446807 20188 tablet_bootstrap.cc:492] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.469081 20188 ts_tablet_manager.cc:1397] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.007s
I20250831 13:57:25.470566 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e0a1ba07754b09a1f73320114c37db" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
I20250831 13:57:25.471431 19999 raft_consensus.cc:2466] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.471364 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e0a1ba07754b09a1f73320114c37db" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.471814 20188 raft_consensus.cc:357] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.472543 20188 raft_consensus.cc:383] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.472826 20188 raft_consensus.cc:738] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.472911 20037 leader_election.cc:304] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.453061 20189 tablet_bootstrap.cc:654] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.474918 20037 leader_election.cc:304] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters: 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:25.446727 20219 raft_consensus.cc:2747] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:25.473524 20188 consensus_queue.cc:260] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.473798 20147 raft_consensus.cc:2466] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.478510 20218 raft_consensus.cc:491] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.474558 20195 raft_consensus.cc:2802] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.479744 20195 raft_consensus.cc:491] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.480137 20195 raft_consensus.cc:3058] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.481699 20219 raft_consensus.cc:2802] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.479593 20218 raft_consensus.cc:513] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.482275 20219 raft_consensus.cc:695] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:25.483167 20219 consensus_queue.cc:237] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.483815 20218 leader_election.cc:290] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.488814 20188 ts_tablet_manager.cc:1428] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.019s user 0.001s sys 0.005s
I20250831 13:57:25.489198 20216 raft_consensus.cc:491] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.489552 20190 tablet_bootstrap.cc:492] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.489673 20216 raft_consensus.cc:513] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.489956 20188 tablet_bootstrap.cc:492] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.489979 20190 ts_tablet_manager.cc:1397] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.067s user 0.019s sys 0.041s
I20250831 13:57:25.492944 20190 raft_consensus.cc:357] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.493588 20190 raft_consensus.cc:383] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.493852 20190 raft_consensus.cc:738] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.494350 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "352060b4fee349d7b694f5bd230d1c0b" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644" is_pre_election: true
I20250831 13:57:25.494516 20190 consensus_queue.cc:260] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.495410 20074 raft_consensus.cc:2466] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 0.
I20250831 13:57:25.496454 19963 leader_election.cc:304] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.497215 20194 raft_consensus.cc:2802] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.497583 20194 raft_consensus.cc:491] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.497752 20190 ts_tablet_manager.cc:1428] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250831 13:57:25.497905 20194 raft_consensus.cc:3058] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.502206 20194 raft_consensus.cc:513] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.502743 20216 leader_election.cc:290] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.505194 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "352060b4fee349d7b694f5bd230d1c0b" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:25.505111 20195 raft_consensus.cc:513] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.505997 20074 raft_consensus.cc:3058] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.507577 20195 leader_election.cc:290] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.512820 20188 tablet_bootstrap.cc:654] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.513269 20196 leader_election.cc:290] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:25.514922 20074 raft_consensus.cc:2466] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 1.
I20250831 13:57:25.516239 19963 leader_election.cc:304] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.517414 20216 raft_consensus.cc:491] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.517948 20216 raft_consensus.cc:513] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.518898 20229 raft_consensus.cc:2802] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.521796 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "352060b4fee349d7b694f5bd230d1c0b" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.522315 20190 tablet_bootstrap.cc:492] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.522374 20147 raft_consensus.cc:2466] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 0.
I20250831 13:57:25.527187 20188 tablet_bootstrap.cc:492] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.527640 20188 ts_tablet_manager.cc:1397] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.000s
I20250831 13:57:25.528913 20190 tablet_bootstrap.cc:654] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.528661 19896 catalog_manager.cc:5582] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.530093 20189 tablet_bootstrap.cc:492] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.530555 20189 ts_tablet_manager.cc:1397] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.104s user 0.020s sys 0.016s
W20250831 13:57:25.530738 20037 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:35961, user_credentials={real_user=slave}} blocked reactor thread for 55851.2us
I20250831 13:57:25.532552 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
I20250831 13:57:25.533288 19999 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.533474 20189 raft_consensus.cc:357] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.534269 20189 raft_consensus.cc:383] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.534560 20189 raft_consensus.cc:738] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.535346 20189 consensus_queue.cc:260] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.535615 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.536329 20147 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.536885 20194 leader_election.cc:290] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.538548 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e0a1ba07754b09a1f73320114c37db" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:25.539037 20216 leader_election.cc:290] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.538992 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644" is_pre_election: true
I20250831 13:57:25.539211 19999 raft_consensus.cc:3058] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.539691 20073 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 0.
I20250831 13:57:25.533946 19894 catalog_manager.cc:5582] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 0 to 1, leader changed from <none> to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 1 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.540517 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.541100 20148 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 0.
I20250831 13:57:25.541270 19963 leader_election.cc:304] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.542588 20216 raft_consensus.cc:2802] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.543107 20216 raft_consensus.cc:491] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.543458 20216 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.545416 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "352060b4fee349d7b694f5bd230d1c0b" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.546082 20147 raft_consensus.cc:3058] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.543530 20189 ts_tablet_manager.cc:1428] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250831 13:57:25.547632 20189 tablet_bootstrap.cc:492] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.562211 20216 raft_consensus.cc:513] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.564839 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:25.565480 20074 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.568092 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
W20250831 13:57:25.569514 20111 leader_election.cc:343] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.571178 19999 raft_consensus.cc:2466] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 1.
I20250831 13:57:25.571856 20074 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 1.
I20250831 13:57:25.572984 19963 leader_election.cc:304] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.573424 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e0a1ba07754b09a1f73320114c37db" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.573743 20194 raft_consensus.cc:2802] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.574095 20148 raft_consensus.cc:3058] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.574489 20194 raft_consensus.cc:695] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:25.575428 20194 consensus_queue.cc:237] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.576787 20189 tablet_bootstrap.cc:654] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.566594 20147 raft_consensus.cc:2466] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 1.
I20250831 13:57:25.580687 20224 raft_consensus.cc:491] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.581317 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644" is_pre_election: true
I20250831 13:57:25.581163 20224 raft_consensus.cc:513] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
W20250831 13:57:25.583307 20112 leader_election.cc:343] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.583725 20112 leader_election.cc:304] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ef2f41fe5e49f3b889122d0efadb19; no voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:25.584540 20224 leader_election.cc:290] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.580065 20229 raft_consensus.cc:695] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:25.584853 20196 raft_consensus.cc:2747] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:25.585693 20148 raft_consensus.cc:2466] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 1.
I20250831 13:57:25.586500 20037 leader_election.cc:304] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:25.587195 20216 leader_election.cc:290] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.530282 20188 raft_consensus.cc:357] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.587792 20188 raft_consensus.cc:383] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.585726 20229 consensus_queue.cc:237] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.588299 20188 raft_consensus.cc:738] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.589416 20190 tablet_bootstrap.cc:492] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.589877 20190 ts_tablet_manager.cc:1397] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.028s
I20250831 13:57:25.590178 20224 raft_consensus.cc:2762] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250831 13:57:25.589720 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.590895 20148 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.591564 20188 consensus_queue.cc:260] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.592649 20190 raft_consensus.cc:357] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.593338 20190 raft_consensus.cc:383] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.593643 20190 raft_consensus.cc:738] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.594362 20190 consensus_queue.cc:260] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.596608 20190 ts_tablet_manager.cc:1428] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250831 13:57:25.596630 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
I20250831 13:57:25.597595 20190 tablet_bootstrap.cc:492] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644: Bootstrap starting.
I20250831 13:57:25.597522 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.598147 19999 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.600018 20188 ts_tablet_manager.cc:1428] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.069s user 0.008s sys 0.000s
I20250831 13:57:25.600806 20188 tablet_bootstrap.cc:492] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.602692 20148 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 1.
I20250831 13:57:25.599066 20147 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.605605 20190 tablet_bootstrap.cc:654] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.599036 20037 leader_election.cc:304] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.606899 20224 raft_consensus.cc:2802] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.607141 20188 tablet_bootstrap.cc:654] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.607627 20224 raft_consensus.cc:695] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:25.608773 20224 consensus_queue.cc:237] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.610162 20219 raft_consensus.cc:491] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.610795 20219 raft_consensus.cc:513] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.613298 20037 leader_election.cc:304] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:25.614269 20219 leader_election.cc:290] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.614987 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
I20250831 13:57:25.614913 20218 raft_consensus.cc:2802] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.615927 20218 raft_consensus.cc:491] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.616380 20218 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.623034 20189 tablet_bootstrap.cc:492] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.623562 20189 ts_tablet_manager.cc:1397] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.076s user 0.020s sys 0.005s
I20250831 13:57:25.626715 20189 raft_consensus.cc:357] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.627476 20189 raft_consensus.cc:383] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.628686 20189 raft_consensus.cc:738] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.629479 20189 consensus_queue.cc:260] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.632066 20189 ts_tablet_manager.cc:1428] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250831 13:57:25.634788 20189 tablet_bootstrap.cc:492] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
I20250831 13:57:25.641899 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
W20250831 13:57:25.643615 20037 leader_election.cc:343] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.618907 19999 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.647301 20189 tablet_bootstrap.cc:654] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.650321 20037 leader_election.cc:304] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters: c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:25.651868 20219 raft_consensus.cc:2802] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.652276 20219 raft_consensus.cc:491] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.652683 20219 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.651199 20218 raft_consensus.cc:513] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.668746 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:25.669528 19999 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.672327 20190 tablet_bootstrap.cc:492] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644: No bootstrap required, opened a new log
I20250831 13:57:25.672992 20190 ts_tablet_manager.cc:1397] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644: Time spent bootstrapping tablet: real 0.076s user 0.020s sys 0.026s
I20250831 13:57:25.676290 20190 raft_consensus.cc:357] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.677136 20190 raft_consensus.cc:383] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.677488 20190 raft_consensus.cc:738] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.678273 20190 consensus_queue.cc:260] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.680876 20190 ts_tablet_manager.cc:1428] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 13:57:25.685699 20188 tablet_bootstrap.cc:492] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.686131 20188 ts_tablet_manager.cc:1397] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.085s user 0.013s sys 0.000s
I20250831 13:57:25.688447 20219 raft_consensus.cc:513] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.688906 20189 tablet_bootstrap.cc:492] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.689458 20189 ts_tablet_manager.cc:1397] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.055s user 0.033s sys 0.006s
I20250831 13:57:25.690332 20219 leader_election.cc:290] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.691342 20218 leader_election.cc:290] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.691395 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.692160 20189 raft_consensus.cc:357] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.692788 20189 raft_consensus.cc:383] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.693092 20189 raft_consensus.cc:738] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.693598 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:25.693256 19894 catalog_manager.cc:5582] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 0 to 1, leader changed from <none> to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 1 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.694216 19998 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.694322 19999 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 1.
I20250831 13:57:25.693773 20189 consensus_queue.cc:260] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.695348 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.695891 20148 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.696645 20189 ts_tablet_manager.cc:1428] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250831 13:57:25.697482 20189 tablet_bootstrap.cc:492] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19: Bootstrap starting.
W20250831 13:57:25.698042 20037 leader_election.cc:343] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:25.699141 19894 catalog_manager.cc:5582] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 0 to 1, leader changed from <none> to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 1 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.700410 20037 leader_election.cc:304] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.701087 20218 raft_consensus.cc:2802] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.701547 20218 raft_consensus.cc:695] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:25.702737 19998 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 1.
I20250831 13:57:25.703182 20218 consensus_queue.cc:237] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.704093 20037 leader_election.cc:304] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters: c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:25.705543 20219 raft_consensus.cc:2802] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.705896 20219 raft_consensus.cc:695] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:25.706876 20189 tablet_bootstrap.cc:654] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.706509 20219 consensus_queue.cc:237] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.714160 20148 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 1.
I20250831 13:57:25.714805 20188 raft_consensus.cc:357] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.715358 20188 raft_consensus.cc:383] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.715641 20188 raft_consensus.cc:738] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.716264 20188 consensus_queue.cc:260] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.718312 20188 ts_tablet_manager.cc:1428] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250831 13:57:25.719477 20188 tablet_bootstrap.cc:492] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.711527 19897 catalog_manager.cc:5582] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.729604 20188 tablet_bootstrap.cc:654] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.737214 20188 tablet_bootstrap.cc:492] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.737536 20188 ts_tablet_manager.cc:1397] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.002s
I20250831 13:57:25.737704 20189 tablet_bootstrap.cc:492] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19: No bootstrap required, opened a new log
I20250831 13:57:25.738108 20189 ts_tablet_manager.cc:1397] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent bootstrapping tablet: real 0.041s user 0.012s sys 0.002s
I20250831 13:57:25.739269 20188 raft_consensus.cc:357] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.739723 20188 raft_consensus.cc:383] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.739915 20188 raft_consensus.cc:738] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.740345 20189 raft_consensus.cc:357] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.740528 20188 consensus_queue.cc:260] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.741057 20189 raft_consensus.cc:383] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.741367 20189 raft_consensus.cc:738] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.741997 20189 consensus_queue.cc:260] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.742748 20188 ts_tablet_manager.cc:1428] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250831 13:57:25.743609 20188 tablet_bootstrap.cc:492] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f: Bootstrap starting.
I20250831 13:57:25.743881 20189 ts_tablet_manager.cc:1428] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250831 13:57:25.749280 20188 tablet_bootstrap.cc:654] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:25.751825 19897 catalog_manager.cc:5582] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.753724 19897 catalog_manager.cc:5582] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.757268 20188 tablet_bootstrap.cc:492] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f: No bootstrap required, opened a new log
I20250831 13:57:25.757722 20188 ts_tablet_manager.cc:1397] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250831 13:57:25.760067 20188 raft_consensus.cc:357] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.760684 20188 raft_consensus.cc:383] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:25.760876 20188 raft_consensus.cc:738] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Initialized, Role: FOLLOWER
I20250831 13:57:25.761420 20188 consensus_queue.cc:260] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.763435 20188 ts_tablet_manager.cc:1428] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250831 13:57:25.773733 20219 raft_consensus.cc:491] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.774080 20219 raft_consensus.cc:513] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.775451 20219 leader_election.cc:290] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.776240 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35588c3345f746f791346a38eb2f2c31" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
I20250831 13:57:25.776458 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35588c3345f746f791346a38eb2f2c31" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.776702 19998 raft_consensus.cc:2466] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.776934 20148 raft_consensus.cc:2466] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 0.
I20250831 13:57:25.777534 20037 leader_election.cc:304] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.778136 20219 raft_consensus.cc:2802] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.778391 20219 raft_consensus.cc:491] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.778633 20219 raft_consensus.cc:3058] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.783063 20219 raft_consensus.cc:513] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.784328 20219 leader_election.cc:290] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.785171 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35588c3345f746f791346a38eb2f2c31" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:25.785274 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35588c3345f746f791346a38eb2f2c31" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.785641 19998 raft_consensus.cc:3058] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.785714 20148 raft_consensus.cc:3058] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.789896 20148 raft_consensus.cc:2466] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 1.
I20250831 13:57:25.790122 19998 raft_consensus.cc:2466] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 1.
I20250831 13:57:25.790714 20037 leader_election.cc:304] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:25.791353 20219 raft_consensus.cc:2802] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.791790 20219 raft_consensus.cc:695] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:25.792490 20219 consensus_queue.cc:237] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.798434 19897 catalog_manager.cc:5582] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.848922 20148 raft_consensus.cc:3058] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.854362 20229 consensus_queue.cc:1035] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:25.861104 20196 raft_consensus.cc:491] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.861568 20196 raft_consensus.cc:513] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.863672 20196 leader_election.cc:290] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:25.865423 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644" is_pre_election: true
I20250831 13:57:25.865825 20229 consensus_queue.cc:1035] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:25.866021 20073 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 0.
I20250831 13:57:25.867218 20112 leader_election.cc:304] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:25.869457 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
I20250831 13:57:25.870157 19998 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 0.
I20250831 13:57:25.872953 20196 raft_consensus.cc:2802] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.873337 20196 raft_consensus.cc:491] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.873689 20196 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.879804 20196 raft_consensus.cc:513] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.881574 20196 leader_election.cc:290] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:25.882318 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:25.882622 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:25.882943 19998 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.883209 20073 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.889703 19998 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 1.
I20250831 13:57:25.889890 20073 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 1.
I20250831 13:57:25.890765 20111 leader_election.cc:304] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998e50cfd9d94f17a96ecbf04b97496f, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:25.891413 20196 raft_consensus.cc:2802] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.891758 20196 raft_consensus.cc:695] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 LEADER]: Becoming Leader. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:25.892482 20196 consensus_queue.cc:237] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.898483 19895 catalog_manager.cc:5582] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:25.913441 20219 consensus_queue.cc:1035] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:25.920990 19998 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.922639 20229 raft_consensus.cc:491] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.923164 20229 raft_consensus.cc:513] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.926468 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644" is_pre_election: true
I20250831 13:57:25.926795 20229 leader_election.cc:290] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.927187 20073 raft_consensus.cc:2466] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 0.
I20250831 13:57:25.927515 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" is_pre_election: true
I20250831 13:57:25.928123 20148 raft_consensus.cc:2466] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 0.
I20250831 13:57:25.928318 19963 leader_election.cc:304] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.929224 20229 raft_consensus.cc:2802] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:25.929570 20229 raft_consensus.cc:491] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:25.929908 20229 raft_consensus.cc:3058] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.929848 20195 consensus_queue.cc:1035] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:25.935559 20229 raft_consensus.cc:513] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.937269 20229 leader_election.cc:290] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:25.937865 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:25.938304 20073 raft_consensus.cc:3058] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.938213 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:25.938841 20148 raft_consensus.cc:3058] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.943768 20073 raft_consensus.cc:2466] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 1.
I20250831 13:57:25.944078 20148 raft_consensus.cc:2466] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 1.
I20250831 13:57:25.944608 19963 leader_election.cc:304] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:25.945328 20229 raft_consensus.cc:2802] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:25.945784 20229 raft_consensus.cc:695] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:25.946605 20229 consensus_queue.cc:237] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.953397 19897 catalog_manager.cc:5582] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 0 to 1, leader changed from <none> to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 1 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:25.967267 20229 consensus_queue.cc:1035] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:25.976503 20229 consensus_queue.cc:1035] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:25.982239 20196 raft_consensus.cc:491] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:25.982663 20196 raft_consensus.cc:513] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:25.984378 20196 leader_election.cc:290] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:25.985368 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f" is_pre_election: true
I20250831 13:57:25.985680 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6544748dcb4a1b88337dd4c55be644" is_pre_election: true
I20250831 13:57:25.986099 19998 raft_consensus.cc:2391] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ef2f41fe5e49f3b889122d0efadb19 in current term 1: Already voted for candidate 0f6544748dcb4a1b88337dd4c55be644 in this term.
I20250831 13:57:25.987100 20112 leader_election.cc:304] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ef2f41fe5e49f3b889122d0efadb19; no voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:25.987994 20196 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:25.994037 20196 raft_consensus.cc:2747] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:26.007589 20229 consensus_queue.cc:1035] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.016872 20216 consensus_queue.cc:1035] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.103859 20195 consensus_queue.cc:1035] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.113652 20219 consensus_queue.cc:1035] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.121915 20219 consensus_queue.cc:1035] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.131742 20219 consensus_queue.cc:1035] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.154213 20073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "627caa15676449a4adc53cbb673ba506"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:35974
I20250831 13:57:26.154732 20073 raft_consensus.cc:604] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:26.159080 20073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:35974
I20250831 13:57:26.159543 20073 raft_consensus.cc:604] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:26.161543 20219 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.164245 20073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4443ba9be133499fb5f093495b583fab"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:35974
I20250831 13:57:26.164678 20073 raft_consensus.cc:604] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:26.172986 20073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f0e0a1ba07754b09a1f73320114c37db"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:35974
I20250831 13:57:26.173697 20073 raft_consensus.cc:604] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:26.174283 20219 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.186322 20218 raft_consensus.cc:991] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:26.186801 20219 raft_consensus.cc:1079] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:26.188084 19999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "627caa15676449a4adc53cbb673ba506"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44568
I20250831 13:57:26.188513 19999 raft_consensus.cc:491] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:26.188809 19999 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.190222 20148 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3"
dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
mode: GRACEFUL
new_leader_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:57554
I20250831 13:57:26.190814 20148 raft_consensus.cc:604] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 LEADER]: Received request to transfer leadership to TS 0f6544748dcb4a1b88337dd4c55be644
I20250831 13:57:26.194415 19999 raft_consensus.cc:513] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.195892 19999 leader_election.cc:290] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:26.196640 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:26.197136 20073 raft_consensus.cc:3053] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:26.197124 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:26.197572 20073 raft_consensus.cc:738] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:26.198413 20073 consensus_queue.cc:260] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.199239 20148 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.199854 20073 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.206388 19999 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:44590
I20250831 13:57:26.206321 20148 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:26.206321 20073 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:26.206982 19999 raft_consensus.cc:604] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Received request to transfer leadership to TS 0f6544748dcb4a1b88337dd4c55be644
I20250831 13:57:26.207623 19963 leader_election.cc:304] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:26.208194 20216 raft_consensus.cc:2802] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:26.208616 20216 raft_consensus.cc:695] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:26.209360 20216 consensus_queue.cc:237] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.212357 19999 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "352060b4fee349d7b694f5bd230d1c0b"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:44590
I20250831 13:57:26.212890 19999 raft_consensus.cc:604] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Received request to transfer leadership to TS 0f6544748dcb4a1b88337dd4c55be644
I20250831 13:57:26.215937 19896 catalog_manager.cc:5582] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 1 to 2, leader changed from 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 2 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:26.217382 19999 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8124d84173894208b877ccf8285f5a82"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:44590
I20250831 13:57:26.217816 19999 raft_consensus.cc:604] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Received request to transfer leadership to TS c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:26.247504 20229 raft_consensus.cc:991] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:26.247865 20216 raft_consensus.cc:1079] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Signalling peer c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:26.249073 20148 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8124d84173894208b877ccf8285f5a82"
dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:57542
I20250831 13:57:26.249557 20148 raft_consensus.cc:491] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:26.249874 20148 raft_consensus.cc:3058] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.254227 20148 raft_consensus.cc:513] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.255612 20148 leader_election.cc:290] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 2 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:26.256350 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8124d84173894208b877ccf8285f5a82" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:26.256703 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8124d84173894208b877ccf8285f5a82" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:26.256891 19999 raft_consensus.cc:3053] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:26.257170 20073 raft_consensus.cc:3058] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.257191 19999 raft_consensus.cc:738] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:26.257891 19999 consensus_queue.cc:260] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.258844 19999 raft_consensus.cc:3058] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.261808 20073 raft_consensus.cc:2466] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 2.
I20250831 13:57:26.262605 20112 leader_election.cc:304] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:26.263262 20196 raft_consensus.cc:2802] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:26.263742 20196 raft_consensus.cc:695] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 LEADER]: Becoming Leader. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:26.263777 19999 raft_consensus.cc:2466] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 2.
I20250831 13:57:26.264659 20196 consensus_queue.cc:237] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.270486 19896 catalog_manager.cc:5582] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 reported cstate change: term changed from 1 to 2, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131). New cstate: current_term: 2 leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:26.310344 20219 consensus_queue.cc:1035] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.319733 20224 consensus_queue.cc:1035] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.343386 20216 consensus_queue.cc:1035] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.352224 20230 consensus_queue.cc:1035] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.360314 20251 consensus_queue.cc:1035] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.365063 20251 consensus_queue.cc:1035] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:26.367322 20196 raft_consensus.cc:991] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19: : Instructing follower 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:26.367887 20251 raft_consensus.cc:1079] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 LEADER]: Signalling peer 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:26.369546 20074 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:35972
I20250831 13:57:26.370441 20074 raft_consensus.cc:491] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:26.370816 20074 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.375922 20229 raft_consensus.cc:991] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:26.376384 20230 raft_consensus.cc:1079] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Signalling peer 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:26.378089 20072 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "352060b4fee349d7b694f5bd230d1c0b"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:35960
I20250831 13:57:26.378623 20072 raft_consensus.cc:491] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:26.378989 20072 raft_consensus.cc:3058] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.381584 20074 raft_consensus.cc:513] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.384191 20074 leader_election.cc:290] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 2 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:26.384152 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:26.384768 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:26.385342 20148 raft_consensus.cc:3053] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:26.384915 19998 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.385655 20148 raft_consensus.cc:738] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:26.386382 20148 consensus_queue.cc:260] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.387279 20148 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.390671 19998 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 2.
I20250831 13:57:26.391716 20037 leader_election.cc:304] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:26.392463 20224 raft_consensus.cc:2802] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:26.392824 20224 raft_consensus.cc:695] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:26.393586 20224 consensus_queue.cc:237] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.396437 20072 raft_consensus.cc:513] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.397668 20148 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 2.
I20250831 13:57:26.397851 20072 leader_election.cc:290] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 2 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:26.399827 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "352060b4fee349d7b694f5bd230d1c0b" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:26.399899 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "352060b4fee349d7b694f5bd230d1c0b" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:26.400292 19998 raft_consensus.cc:3053] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:26.400461 20148 raft_consensus.cc:3058] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.400614 19998 raft_consensus.cc:738] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:26.401741 19998 consensus_queue.cc:260] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.402070 19897 catalog_manager.cc:5582] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 1 to 2, leader changed from c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131) to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:26.403041 19998 raft_consensus.cc:3058] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.406714 20218 raft_consensus.cc:991] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:26.406733 20148 raft_consensus.cc:2466] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 2.
I20250831 13:57:26.407217 20218 raft_consensus.cc:1079] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:26.408458 20037 leader_election.cc:304] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:26.408529 19999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44568
I20250831 13:57:26.409160 19999 raft_consensus.cc:491] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:26.409407 19999 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.409418 19998 raft_consensus.cc:2466] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 2.
I20250831 13:57:26.414044 20224 raft_consensus.cc:2802] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:26.413903 19999 raft_consensus.cc:513] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.414492 20224 raft_consensus.cc:695] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:26.415328 20224 consensus_queue.cc:237] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.416834 19999 leader_election.cc:290] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:26.417718 20072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:26.418265 20072 raft_consensus.cc:3053] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:26.418201 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:26.418656 20072 raft_consensus.cc:738] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:26.418839 20148 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.419430 20072 consensus_queue.cc:260] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.420540 20072 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.421617 19897 catalog_manager.cc:5582] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 1 to 2, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:26.424978 20148 raft_consensus.cc:2466] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:26.425941 19962 leader_election.cc:304] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998e50cfd9d94f17a96ecbf04b97496f, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:26.426736 20230 raft_consensus.cc:2802] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:26.426908 20072 raft_consensus.cc:2466] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:26.427256 20230 raft_consensus.cc:695] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:26.428054 20230 consensus_queue.cc:237] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.434193 19897 catalog_manager.cc:5582] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 1 to 2, leader changed from 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 2 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:26.602639 20219 raft_consensus.cc:991] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:26.603068 20219 raft_consensus.cc:1079] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:26.604365 19999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4443ba9be133499fb5f093495b583fab"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44568
I20250831 13:57:26.604797 19999 raft_consensus.cc:491] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:26.605082 19999 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.609553 19999 raft_consensus.cc:513] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.610478 20072 raft_consensus.cc:1273] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.611083 19999 leader_election.cc:290] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:26.611892 20230 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.612075 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:26.612094 20072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:26.612627 20148 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.612799 20072 raft_consensus.cc:3053] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:26.613178 20072 raft_consensus.cc:738] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:26.614022 20072 consensus_queue.cc:260] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.615214 20072 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.620436 20147 raft_consensus.cc:1273] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.620769 20148 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:26.621845 20216 raft_consensus.cc:991] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:26.621966 20072 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:26.622076 19962 leader_election.cc:304] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998e50cfd9d94f17a96ecbf04b97496f, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:26.623773 20216 raft_consensus.cc:1079] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Signalling peer 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:26.623685 20229 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.624200 20279 raft_consensus.cc:2802] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:26.624706 20279 raft_consensus.cc:695] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:26.625579 20074 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:35960
I20250831 13:57:26.626242 20074 raft_consensus.cc:491] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:26.626828 20074 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.634146 20074 raft_consensus.cc:513] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.625586 20279 consensus_queue.cc:237] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.636140 20074 leader_election.cc:290] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 2 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:26.637521 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:26.638178 19999 raft_consensus.cc:3053] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:26.638559 19999 raft_consensus.cc:738] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:26.640132 19999 consensus_queue.cc:260] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.641410 19999 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.641505 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:26.642058 20148 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.647827 20148 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 2.
I20250831 13:57:26.647733 19999 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 2.
I20250831 13:57:26.648772 20037 leader_election.cc:304] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:26.649570 20219 raft_consensus.cc:2802] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:26.650068 20219 raft_consensus.cc:695] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:26.649459 19896 catalog_manager.cc:5582] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 1 to 2, leader changed from 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 2 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:26.650926 20219 consensus_queue.cc:237] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.657436 19897 catalog_manager.cc:5582] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 1 to 2, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:26.670981 20219 raft_consensus.cc:991] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:26.671406 20224 raft_consensus.cc:1079] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:26.672642 19999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f0e0a1ba07754b09a1f73320114c37db"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44568
I20250831 13:57:26.673062 19999 raft_consensus.cc:491] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:26.673416 19999 raft_consensus.cc:3058] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.677641 19999 raft_consensus.cc:513] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.679104 19999 leader_election.cc:290] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:26.679755 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e0a1ba07754b09a1f73320114c37db" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:26.680114 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f0e0a1ba07754b09a1f73320114c37db" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:26.680215 20074 raft_consensus.cc:3053] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:26.680496 20074 raft_consensus.cc:738] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:26.680516 20148 raft_consensus.cc:3058] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.681126 20074 consensus_queue.cc:260] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.681955 20074 raft_consensus.cc:3058] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:26.684616 20148 raft_consensus.cc:2466] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:26.685460 19962 leader_election.cc:304] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998e50cfd9d94f17a96ecbf04b97496f, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:26.686053 20279 raft_consensus.cc:2802] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:26.686213 20074 raft_consensus.cc:2466] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:26.686419 20279 raft_consensus.cc:695] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:26.687191 20279 consensus_queue.cc:237] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:26.692731 19897 catalog_manager.cc:5582] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 1 to 2, leader changed from 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 2 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:26.803735 20074 raft_consensus.cc:1273] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.804989 20196 consensus_queue.cc:1035] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.812391 19999 raft_consensus.cc:1273] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.813699 20196 consensus_queue.cc:1035] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.859916 20148 raft_consensus.cc:1273] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.861246 20224 consensus_queue.cc:1035] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.867973 19999 raft_consensus.cc:1273] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.869597 20224 consensus_queue.cc:1035] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.919214 20148 raft_consensus.cc:1273] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.920303 20224 consensus_queue.cc:1035] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.927668 19999 raft_consensus.cc:1273] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.928992 20218 consensus_queue.cc:1035] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.990916 20147 raft_consensus.cc:1273] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:26.992069 20279 consensus_queue.cc:1035] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:26.999615 20074 raft_consensus.cc:1273] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:27.001080 20279 consensus_queue.cc:1035] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:27.038833 20147 raft_consensus.cc:1273] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:27.039975 20219 consensus_queue.cc:1035] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:27.047705 19998 raft_consensus.cc:1273] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:27.049186 20224 consensus_queue.cc:1035] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:27.113533 20147 raft_consensus.cc:1273] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:27.114749 20216 consensus_queue.cc:1035] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:27.121603 20074 raft_consensus.cc:1273] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:27.122710 20072 raft_consensus.cc:1273] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:27.123183 20216 consensus_queue.cc:1035] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:57:27.123893 20279 consensus_queue.cc:1035] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:27.133548 20147 raft_consensus.cc:1273] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:27.135562 20229 consensus_queue.cc:1035] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:29.245730 19366 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20250831 13:57:29.246337 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:29.261009 19998 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4443ba9be133499fb5f093495b583fab"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:44592
I20250831 13:57:29.261507 19998 raft_consensus.cc:604] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Received request to transfer leadership to TS c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:29.263509 19998 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "627caa15676449a4adc53cbb673ba506"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:44592
I20250831 13:57:29.263911 19998 raft_consensus.cc:604] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Received request to transfer leadership to TS c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:29.264801 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250831 13:57:29.265177 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:29.328365 20229 raft_consensus.cc:991] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:29.328797 20290 raft_consensus.cc:1079] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Signalling peer c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:29.330082 20147 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4443ba9be133499fb5f093495b583fab"
dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:57542
I20250831 13:57:29.330547 20147 raft_consensus.cc:491] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:29.330927 20147 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:29.335665 20147 raft_consensus.cc:513] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:29.337119 20147 leader_election.cc:290] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 3 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:29.338090 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:29.337846 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:29.338613 20073 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:29.338771 19998 raft_consensus.cc:3053] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:29.339125 19998 raft_consensus.cc:738] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:29.339900 19998 consensus_queue.cc:260] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:29.341017 19998 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:29.345659 20073 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 3.
I20250831 13:57:29.346879 20112 leader_election.cc:304] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:29.347679 20196 raft_consensus.cc:2802] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:29.347891 19998 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 3.
I20250831 13:57:29.348634 20196 raft_consensus.cc:695] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Becoming Leader. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:29.349491 20196 consensus_queue.cc:237] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:29.356782 19896 catalog_manager.cc:5582] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 reported cstate change: term changed from 2 to 3, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:29.361214 20229 raft_consensus.cc:991] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:29.361626 20290 raft_consensus.cc:1079] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Signalling peer c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:29.363240 20147 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "627caa15676449a4adc53cbb673ba506"
dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:57542
I20250831 13:57:29.363713 20147 raft_consensus.cc:491] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:29.363986 20147 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:29.369686 20147 raft_consensus.cc:513] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:29.371210 20147 leader_election.cc:290] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 3 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:29.371829 19998 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:29.372313 19998 raft_consensus.cc:3053] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:29.372273 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:29.372682 19998 raft_consensus.cc:738] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:29.372875 20073 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:29.373351 19998 consensus_queue.cc:260] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:29.374217 19998 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:29.377231 20073 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 3.
I20250831 13:57:29.378168 20112 leader_election.cc:304] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:29.378580 19998 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 3.
I20250831 13:57:29.378809 20196 raft_consensus.cc:2802] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:29.379261 20196 raft_consensus.cc:695] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Becoming Leader. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:29.379964 20196 consensus_queue.cc:237] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:29.391146 19896 catalog_manager.cc:5582] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 reported cstate change: term changed from 2 to 3, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:29.873901 20072 raft_consensus.cc:1273] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:29.875170 20196 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:29.883349 19998 raft_consensus.cc:1273] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:29.885773 20291 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:29.910847 20072 raft_consensus.cc:1273] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:29.911939 20196 consensus_queue.cc:1035] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:29.919471 19998 raft_consensus.cc:1273] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:29.920847 20304 consensus_queue.cc:1035] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:30.265955 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:30.273241 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:30.273598 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:31.274421 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:31.279511 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:31.279839 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:32.280577 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:32.285787 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:32.286123 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:33.287026 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:33.292711 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:33.293129 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:34.294327 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:34.300586 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:34.300959 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:35.302011 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:35.307019 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:35.307443 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:36.308347 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:36.313257 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:36.313578 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:37.314342 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:37.319382 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:37.319711 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:38.320421 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:38.325353 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:38.325670 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:39.351784 19366 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20250831 13:57:39.355787 20147 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "627caa15676449a4adc53cbb673ba506"
dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:57554
I20250831 13:57:39.356344 20147 raft_consensus.cc:604] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:39.362244 20147 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4443ba9be133499fb5f093495b583fab"
dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:57554
I20250831 13:57:39.362636 20147 raft_consensus.cc:604] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:39.370982 20073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:35974
I20250831 13:57:39.371398 20073 raft_consensus.cc:604] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:39.375285 20073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "35588c3345f746f791346a38eb2f2c31"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:35974
I20250831 13:57:39.375622 20073 raft_consensus.cc:604] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:39.379773 20073 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
mode: GRACEFUL
new_leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:35974
I20250831 13:57:39.380139 20073 raft_consensus.cc:604] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Received request to transfer leadership to TS 998e50cfd9d94f17a96ecbf04b97496f
I20250831 13:57:39.455598 20307 raft_consensus.cc:991] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.456173 20314 raft_consensus.cc:991] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.456218 20313 raft_consensus.cc:1079] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.456698 20314 raft_consensus.cc:1079] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.458101 19998 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44568
I20250831 13:57:39.458623 19998 raft_consensus.cc:491] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:39.458555 19999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44568
I20250831 13:57:39.459266 19998 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:39.459414 19999 raft_consensus.cc:491] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:39.459933 19999 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:39.466626 19998 raft_consensus.cc:513] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.466933 19999 raft_consensus.cc:513] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.468338 19998 leader_election.cc:290] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 3 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:39.468485 19999 leader_election.cc:290] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 3 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:39.469746 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:39.470456 20073 raft_consensus.cc:3053] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:39.470561 20148 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:39.471235 20148 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:39.471498 20072 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:39.472005 20072 raft_consensus.cc:3053] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:39.472265 20072 raft_consensus.cc:738] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:39.473044 20072 consensus_queue.cc:260] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.474191 20072 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:39.470837 20073 raft_consensus.cc:738] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:39.475296 20073 consensus_queue.cc:260] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.476097 20073 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:39.478477 20148 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 3.
I20250831 13:57:39.479049 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae2e10d60e53469396554f7ff859aa48" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:39.479532 20147 raft_consensus.cc:3058] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:39.479936 19962 leader_election.cc:304] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998e50cfd9d94f17a96ecbf04b97496f, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:39.484422 20312 raft_consensus.cc:2802] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:39.484966 20312 raft_consensus.cc:695] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:39.485971 20072 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 3.
I20250831 13:57:39.486014 20312 consensus_queue.cc:237] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.492478 20073 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 3.
I20250831 13:57:39.496150 20147 raft_consensus.cc:2466] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 3.
I20250831 13:57:39.498915 19962 leader_election.cc:304] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998e50cfd9d94f17a96ecbf04b97496f, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:39.500088 20315 raft_consensus.cc:2802] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:39.500394 19897 catalog_manager.cc:5582] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 2 to 3, leader changed from 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 3 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:39.501854 20315 raft_consensus.cc:695] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:39.502771 20315 consensus_queue.cc:237] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.517762 19897 catalog_manager.cc:5582] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 2 to 3, leader changed from 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 3 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:39.697935 20309 raft_consensus.cc:991] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.698395 20324 raft_consensus.cc:1079] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.699656 19999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4443ba9be133499fb5f093495b583fab"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44584
I20250831 13:57:39.700114 19999 raft_consensus.cc:491] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:39.700412 19999 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:39.705847 19999 raft_consensus.cc:513] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.707610 19999 leader_election.cc:290] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 4 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:39.708372 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:39.708781 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4443ba9be133499fb5f093495b583fab" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:39.708921 20073 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:39.709280 20147 raft_consensus.cc:3053] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:57:39.709591 20147 raft_consensus.cc:738] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:39.710096 20147 consensus_queue.cc:260] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.710938 20147 raft_consensus.cc:3058] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:39.712859 20073 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 4.
I20250831 13:57:39.713793 19963 leader_election.cc:304] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:39.714355 20315 raft_consensus.cc:2802] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:57:39.714434 20147 raft_consensus.cc:2466] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 4.
I20250831 13:57:39.714727 20315 raft_consensus.cc:695] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 4 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:39.715484 20315 consensus_queue.cc:237] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.721915 19897 catalog_manager.cc:5582] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 3 to 4, leader changed from c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 4 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:39.781949 20313 raft_consensus.cc:991] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.782341 20314 raft_consensus.cc:1079] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.783697 19999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "35588c3345f746f791346a38eb2f2c31"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44568
I20250831 13:57:39.784116 19999 raft_consensus.cc:491] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:39.784402 19999 raft_consensus.cc:3058] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:39.789125 19999 raft_consensus.cc:513] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.790584 19999 leader_election.cc:290] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:39.791379 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35588c3345f746f791346a38eb2f2c31" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:39.791575 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "35588c3345f746f791346a38eb2f2c31" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:39.791879 20073 raft_consensus.cc:3053] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:39.792022 20147 raft_consensus.cc:3058] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:39.792150 20073 raft_consensus.cc:738] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:39.792836 20073 consensus_queue.cc:260] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.793679 20073 raft_consensus.cc:3058] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:39.795878 20147 raft_consensus.cc:2466] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:39.796579 19962 leader_election.cc:304] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 998e50cfd9d94f17a96ecbf04b97496f, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:39.797165 20315 raft_consensus.cc:2802] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:39.797587 20315 raft_consensus.cc:695] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:39.798454 20073 raft_consensus.cc:2466] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 2.
I20250831 13:57:39.798444 20315 consensus_queue.cc:237] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.804615 19897 catalog_manager.cc:5582] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 1 to 2, leader changed from 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 2 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:39.844976 20309 raft_consensus.cc:991] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19: : Instructing follower 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.845358 20324 raft_consensus.cc:1079] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Signalling peer 998e50cfd9d94f17a96ecbf04b97496f to start an election
I20250831 13:57:39.846693 19999 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "627caa15676449a4adc53cbb673ba506"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
from {username='slave'} at 127.0.0.1:44584
I20250831 13:57:39.847136 19999 raft_consensus.cc:491] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:39.847410 19999 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:39.851434 19999 raft_consensus.cc:513] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.852666 19999 leader_election.cc:290] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 4 election: Requested vote from peers 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:39.853390 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:39.853623 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "998e50cfd9d94f17a96ecbf04b97496f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:39.853902 20073 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:39.854077 20147 raft_consensus.cc:3053] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:57:39.854363 20147 raft_consensus.cc:738] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:39.855099 20147 consensus_queue.cc:260] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.856191 20147 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:39.859926 20073 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 4.
I20250831 13:57:39.860872 19963 leader_election.cc:304] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, 998e50cfd9d94f17a96ecbf04b97496f; no voters:
I20250831 13:57:39.861460 20315 raft_consensus.cc:2802] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:57:39.861820 20147 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 998e50cfd9d94f17a96ecbf04b97496f in term 4.
I20250831 13:57:39.861954 20315 raft_consensus.cc:695] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 LEADER]: Becoming Leader. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:39.862874 20315 consensus_queue.cc:237] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:39.869604 19897 catalog_manager.cc:5582] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f reported cstate change: term changed from 3 to 4, leader changed from c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131) to 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129). New cstate: current_term: 4 leader_uuid: "998e50cfd9d94f17a96ecbf04b97496f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:39.925930 20073 raft_consensus.cc:1273] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:39.927176 20318 consensus_queue.cc:1035] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:57:39.928231 20147 raft_consensus.cc:1273] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:39.929335 20318 consensus_queue.cc:1035] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:40.078234 20147 raft_consensus.cc:1273] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:40.079236 20315 consensus_queue.cc:1035] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:40.086768 20073 raft_consensus.cc:1273] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:40.088023 20315 consensus_queue.cc:1035] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:40.214382 20073 raft_consensus.cc:1273] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:40.215368 20315 consensus_queue.cc:1035] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:40.223426 20147 raft_consensus.cc:1273] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:40.224862 20312 consensus_queue.cc:1035] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:40.342330 20147 raft_consensus.cc:1273] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:40.343549 20312 consensus_queue.cc:1035] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:40.351210 20073 raft_consensus.cc:1273] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:40.352957 20312 consensus_queue.cc:1035] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:40.384608 20073 raft_consensus.cc:1273] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:40.385576 20312 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:40.392724 20147 raft_consensus.cc:1273] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Refusing update from remote peer 998e50cfd9d94f17a96ecbf04b97496f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:40.394033 20316 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:42.411173 19366 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20250831 13:57:42.411934 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:42.419889 19999 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:44592
I20250831 13:57:42.420300 19999 raft_consensus.cc:604] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Received request to transfer leadership to TS c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:42.422220 19999 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:44592
I20250831 13:57:42.422719 19999 raft_consensus.cc:604] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 LEADER]: Received request to transfer leadership to TS c5ef2f41fe5e49f3b889122d0efadb19
I20250831 13:57:42.424827 19999 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:44592
I20250831 13:57:42.425202 19999 raft_consensus.cc:604] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Received request to transfer leadership to TS 0f6544748dcb4a1b88337dd4c55be644
I20250831 13:57:42.426920 19999 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "627caa15676449a4adc53cbb673ba506"
dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
mode: GRACEFUL
new_leader_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:44592
I20250831 13:57:42.427291 19999 raft_consensus.cc:604] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 LEADER]: Received request to transfer leadership to TS 0f6544748dcb4a1b88337dd4c55be644
I20250831 13:57:42.428135 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250831 13:57:42.428654 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:42.435247 20315 raft_consensus.cc:991] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:42.435626 20312 raft_consensus.cc:1079] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 LEADER]: Signalling peer 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:42.436861 20074 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "627caa15676449a4adc53cbb673ba506"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:35960
I20250831 13:57:42.437242 20074 raft_consensus.cc:491] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:42.437520 20074 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:42.442153 20074 raft_consensus.cc:513] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.443984 20074 leader_election.cc:290] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 5 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:42.444739 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:42.444921 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "627caa15676449a4adc53cbb673ba506" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:42.445204 19999 raft_consensus.cc:3053] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:57:42.445353 20147 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:42.445470 19999 raft_consensus.cc:738] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:42.446058 19999 consensus_queue.cc:260] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.446847 19999 raft_consensus.cc:3058] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:57:42.449406 20147 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 5.
I20250831 13:57:42.450352 20037 leader_election.cc:304] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:42.450971 20314 raft_consensus.cc:2802] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:57:42.451032 19999 raft_consensus.cc:2466] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 5.
I20250831 13:57:42.452062 20314 raft_consensus.cc:695] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 5 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:42.452824 20314 consensus_queue.cc:237] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.458392 19897 catalog_manager.cc:5582] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 4 to 5, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 5 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:42.560530 20315 raft_consensus.cc:991] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:42.560981 20312 raft_consensus.cc:1079] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Signalling peer 0f6544748dcb4a1b88337dd4c55be644 to start an election
I20250831 13:57:42.562170 20074 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372"
dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
from {username='slave'} at 127.0.0.1:35960
I20250831 13:57:42.562639 20074 raft_consensus.cc:491] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:42.562945 20074 raft_consensus.cc:3058] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:42.567386 20074 raft_consensus.cc:513] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.568760 20074 leader_election.cc:290] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 2 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131:35961)
I20250831 13:57:42.569545 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:42.569684 20147 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "274f624b6e8d4c0ebbcf1bafbf5e5372" candidate_uuid: "0f6544748dcb4a1b88337dd4c55be644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
I20250831 13:57:42.570068 19999 raft_consensus.cc:3053] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:42.570173 20147 raft_consensus.cc:3058] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:42.570341 19999 raft_consensus.cc:738] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:42.571134 19999 consensus_queue.cc:260] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.572269 19999 raft_consensus.cc:3058] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:42.574738 20147 raft_consensus.cc:2466] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 2.
I20250831 13:57:42.575500 20037 leader_election.cc:304] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:42.576069 20314 raft_consensus.cc:2802] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:57:42.576256 19999 raft_consensus.cc:2466] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f6544748dcb4a1b88337dd4c55be644 in term 2.
I20250831 13:57:42.576427 20314 raft_consensus.cc:695] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Becoming Leader. State: Replica: 0f6544748dcb4a1b88337dd4c55be644, State: Running, Role: LEADER
I20250831 13:57:42.577297 20314 consensus_queue.cc:237] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.583702 19897 catalog_manager.cc:5582] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 reported cstate change: term changed from 1 to 2, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "0f6544748dcb4a1b88337dd4c55be644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:42.767343 20315 raft_consensus.cc:991] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:42.767782 20312 raft_consensus.cc:1079] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Signalling peer c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:42.769088 20147 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258"
dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:57542
I20250831 13:57:42.769501 20147 raft_consensus.cc:491] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:42.769753 20147 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:42.773715 20147 raft_consensus.cc:513] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.775463 20147 leader_election.cc:290] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 3 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:42.776278 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:42.776544 20074 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01a91fa016854945a8cd940e2af5f258" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:42.776743 19999 raft_consensus.cc:3053] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:57:42.777017 19999 raft_consensus.cc:738] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:42.777045 20074 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:42.777608 19999 consensus_queue.cc:260] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.778654 19999 raft_consensus.cc:3058] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:42.783268 20074 raft_consensus.cc:2466] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 3.
I20250831 13:57:42.783967 19999 raft_consensus.cc:2466] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 3.
I20250831 13:57:42.784459 20112 leader_election.cc:304] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:42.785221 20309 raft_consensus.cc:2802] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:42.786377 20309 raft_consensus.cc:695] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Becoming Leader. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:42.787097 20309 consensus_queue.cc:237] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.793753 19897 catalog_manager.cc:5582] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 reported cstate change: term changed from 2 to 3, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:42.835877 20147 raft_consensus.cc:1273] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 5 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 13:57:42.836920 20314 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:57:42.846668 19999 raft_consensus.cc:1273] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 5 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 13:57:42.848373 20314 consensus_queue.cc:1035] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:57:42.872470 20315 raft_consensus.cc:991] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f: : Instructing follower c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:42.872891 20312 raft_consensus.cc:1079] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 LEADER]: Signalling peer c5ef2f41fe5e49f3b889122d0efadb19 to start an election
I20250831 13:57:42.874135 20147 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3"
dest_uuid: "c5ef2f41fe5e49f3b889122d0efadb19"
from {username='slave'} at 127.0.0.1:57542
I20250831 13:57:42.874493 20147 raft_consensus.cc:491] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:57:42.874751 20147 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:42.878521 20147 raft_consensus.cc:513] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.879701 20147 leader_election.cc:290] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 4 election: Requested vote from peers 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129:37799), 0f6544748dcb4a1b88337dd4c55be644 (127.18.233.130:37829)
I20250831 13:57:42.880481 19999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "998e50cfd9d94f17a96ecbf04b97496f"
I20250831 13:57:42.880650 20073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "096f3a0441b04af0b9352bf7ef8b42d3" candidate_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "0f6544748dcb4a1b88337dd4c55be644"
I20250831 13:57:42.881023 19999 raft_consensus.cc:3053] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:57:42.881088 20073 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:42.881377 19999 raft_consensus.cc:738] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 998e50cfd9d94f17a96ecbf04b97496f, State: Running, Role: LEADER
I20250831 13:57:42.882061 19999 consensus_queue.cc:260] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.883134 19999 raft_consensus.cc:3058] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:57:42.887094 20073 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 4.
I20250831 13:57:42.888005 20112 leader_election.cc:304] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6544748dcb4a1b88337dd4c55be644, c5ef2f41fe5e49f3b889122d0efadb19; no voters:
I20250831 13:57:42.888758 20309 raft_consensus.cc:2802] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:57:42.889065 19999 raft_consensus.cc:2466] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ef2f41fe5e49f3b889122d0efadb19 in term 4.
I20250831 13:57:42.889125 20309 raft_consensus.cc:695] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 LEADER]: Becoming Leader. State: Replica: c5ef2f41fe5e49f3b889122d0efadb19, State: Running, Role: LEADER
I20250831 13:57:42.889974 20309 consensus_queue.cc:237] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } }
I20250831 13:57:42.896055 19896 catalog_manager.cc:5582] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 reported cstate change: term changed from 3 to 4, leader changed from 998e50cfd9d94f17a96ecbf04b97496f (127.18.233.129) to c5ef2f41fe5e49f3b889122d0efadb19 (127.18.233.131). New cstate: current_term: 4 leader_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:43.018338 19999 raft_consensus.cc:1273] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:43.019457 20314 consensus_queue.cc:1035] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:43.029105 20147 raft_consensus.cc:1273] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Refusing update from remote peer 0f6544748dcb4a1b88337dd4c55be644: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:57:43.030309 20314 consensus_queue.cc:1035] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ef2f41fe5e49f3b889122d0efadb19" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 35961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:57:43.280999 19999 raft_consensus.cc:1273] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:43.281991 20309 consensus_queue.cc:1035] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:43.289784 20073 raft_consensus.cc:1273] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:57:43.292461 20309 consensus_queue.cc:1035] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:57:43.385190 20073 raft_consensus.cc:1273] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:43.386832 20309 consensus_queue.cc:1035] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6544748dcb4a1b88337dd4c55be644" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 37829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:43.398993 19999 raft_consensus.cc:1273] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Refusing update from remote peer c5ef2f41fe5e49f3b889122d0efadb19: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:57:43.401556 20348 consensus_queue.cc:1035] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "998e50cfd9d94f17a96ecbf04b97496f" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 37799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:57:43.429450 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:43.434806 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:43.435185 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:44.436163 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:44.443558 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:44.443969 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:45.444869 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:45.450008 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:45.450388 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:46.451200 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:46.456236 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:46.456648 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:47.457427 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:47.463889 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:47.464417 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:48.465247 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:48.470001 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:48.470386 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:49.471261 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:49.476197 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:49.476509 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:50.477339 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:50.482131 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:50.482450 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:51.483240 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:57:51.488303 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250831 13:57:51.488623 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:57:52.514112 19366 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20250831 13:57:52.514562 19366 tablet_server.cc:178] TabletServer@127.18.233.129:0 shutting down...
I20250831 13:57:52.539410 19366 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 13:57:52.540042 19366 tablet_replica.cc:331] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.540623 19366 raft_consensus.cc:2241] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.541110 19366 raft_consensus.cc:2270] T 01a91fa016854945a8cd940e2af5f258 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.542939 19366 tablet_replica.cc:331] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.543462 19366 raft_consensus.cc:2241] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.543859 19366 raft_consensus.cc:2270] T 627caa15676449a4adc53cbb673ba506 P 998e50cfd9d94f17a96ecbf04b97496f [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.545374 19366 tablet_replica.cc:331] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.545816 19366 raft_consensus.cc:2241] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.546173 19366 raft_consensus.cc:2270] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.547780 19366 tablet_replica.cc:331] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.548224 19366 raft_consensus.cc:2241] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.548923 19366 raft_consensus.cc:2270] T 35588c3345f746f791346a38eb2f2c31 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.550474 19366 tablet_replica.cc:331] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.551036 19366 raft_consensus.cc:2241] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.551400 19366 raft_consensus.cc:2270] T 8124d84173894208b877ccf8285f5a82 P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.552860 19366 tablet_replica.cc:331] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.553259 19366 raft_consensus.cc:2241] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.553599 19366 raft_consensus.cc:2270] T 096f3a0441b04af0b9352bf7ef8b42d3 P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.555115 19366 tablet_replica.cc:331] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.555523 19366 raft_consensus.cc:2241] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 4 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.556360 19366 raft_consensus.cc:2270] T 4443ba9be133499fb5f093495b583fab P 998e50cfd9d94f17a96ecbf04b97496f [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.558415 19366 tablet_replica.cc:331] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.559017 19366 raft_consensus.cc:2241] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 2 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.559763 19366 raft_consensus.cc:2270] T f0e0a1ba07754b09a1f73320114c37db P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.561352 19366 tablet_replica.cc:331] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.561812 19366 raft_consensus.cc:2241] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.562315 19366 raft_consensus.cc:2270] T 352060b4fee349d7b694f5bd230d1c0b P 998e50cfd9d94f17a96ecbf04b97496f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.564193 19366 tablet_replica.cc:331] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f: stopping tablet replica
I20250831 13:57:52.564558 19366 raft_consensus.cc:2241] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.565236 19366 raft_consensus.cc:2270] T ae2e10d60e53469396554f7ff859aa48 P 998e50cfd9d94f17a96ecbf04b97496f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.594285 19366 tablet_server.cc:195] TabletServer@127.18.233.129:0 shutdown complete.
I20250831 13:57:52.612550 19366 tablet_server.cc:178] TabletServer@127.18.233.130:0 shutting down...
I20250831 13:57:52.637781 19366 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 13:57:52.638523 19366 tablet_replica.cc:331] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.639537 19366 raft_consensus.cc:2241] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.640268 19366 raft_consensus.cc:2270] T 01a91fa016854945a8cd940e2af5f258 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.642177 19366 tablet_replica.cc:331] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.642704 19366 raft_consensus.cc:2241] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.643461 19366 raft_consensus.cc:2270] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.645293 19366 tablet_replica.cc:331] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.645790 19366 raft_consensus.cc:2241] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.646162 19366 raft_consensus.cc:2270] T 35588c3345f746f791346a38eb2f2c31 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.647851 19366 tablet_replica.cc:331] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.648344 19366 raft_consensus.cc:2241] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.648694 19366 raft_consensus.cc:2270] T 8124d84173894208b877ccf8285f5a82 P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.650177 19366 tablet_replica.cc:331] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.650592 19366 raft_consensus.cc:2241] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 5 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.651392 19366 raft_consensus.cc:2270] T 627caa15676449a4adc53cbb673ba506 P 0f6544748dcb4a1b88337dd4c55be644 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.653070 19366 tablet_replica.cc:331] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.653494 19366 raft_consensus.cc:2241] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.653858 19366 raft_consensus.cc:2270] T f0e0a1ba07754b09a1f73320114c37db P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.655579 19366 tablet_replica.cc:331] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.655994 19366 raft_consensus.cc:2241] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.656343 19366 raft_consensus.cc:2270] T 096f3a0441b04af0b9352bf7ef8b42d3 P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.658022 19366 tablet_replica.cc:331] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.658427 19366 raft_consensus.cc:2241] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.658890 19366 raft_consensus.cc:2270] T 4443ba9be133499fb5f093495b583fab P 0f6544748dcb4a1b88337dd4c55be644 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.660646 19366 tablet_replica.cc:331] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.661149 19366 raft_consensus.cc:2241] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 2 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.661867 19366 raft_consensus.cc:2270] T 352060b4fee349d7b694f5bd230d1c0b P 0f6544748dcb4a1b88337dd4c55be644 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.663565 19366 tablet_replica.cc:331] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644: stopping tablet replica
I20250831 13:57:52.664000 19366 raft_consensus.cc:2241] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.664393 19366 raft_consensus.cc:2270] T ae2e10d60e53469396554f7ff859aa48 P 0f6544748dcb4a1b88337dd4c55be644 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.692848 19366 tablet_server.cc:195] TabletServer@127.18.233.130:0 shutdown complete.
I20250831 13:57:52.709764 19366 tablet_server.cc:178] TabletServer@127.18.233.131:0 shutting down...
I20250831 13:57:52.739012 19366 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250831 13:57:52.739558 19366 tablet_replica.cc:331] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.740154 19366 raft_consensus.cc:2241] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.740628 19366 raft_consensus.cc:2270] T 627caa15676449a4adc53cbb673ba506 P c5ef2f41fe5e49f3b889122d0efadb19 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.742410 19366 tablet_replica.cc:331] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.742929 19366 raft_consensus.cc:2241] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.743669 19366 raft_consensus.cc:2270] T 096f3a0441b04af0b9352bf7ef8b42d3 P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.745407 19366 tablet_replica.cc:331] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.745874 19366 raft_consensus.cc:2241] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.746276 19366 raft_consensus.cc:2270] T 4443ba9be133499fb5f093495b583fab P c5ef2f41fe5e49f3b889122d0efadb19 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.747790 19366 tablet_replica.cc:331] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.748251 19366 raft_consensus.cc:2241] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.748871 19366 raft_consensus.cc:2270] T 8124d84173894208b877ccf8285f5a82 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.750391 19366 tablet_replica.cc:331] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.750967 19366 raft_consensus.cc:2241] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.751787 19366 raft_consensus.cc:2270] T 274f624b6e8d4c0ebbcf1bafbf5e5372 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.754014 19366 tablet_replica.cc:331] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.754462 19366 raft_consensus.cc:2241] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.754877 19366 raft_consensus.cc:2270] T 35588c3345f746f791346a38eb2f2c31 P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.756352 19366 tablet_replica.cc:331] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.756765 19366 raft_consensus.cc:2241] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.757126 19366 raft_consensus.cc:2270] T f0e0a1ba07754b09a1f73320114c37db P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.758631 19366 tablet_replica.cc:331] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.759195 19366 raft_consensus.cc:2241] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.759891 19366 raft_consensus.cc:2270] T 01a91fa016854945a8cd940e2af5f258 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.761595 19366 tablet_replica.cc:331] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.762032 19366 raft_consensus.cc:2241] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.762430 19366 raft_consensus.cc:2270] T 352060b4fee349d7b694f5bd230d1c0b P c5ef2f41fe5e49f3b889122d0efadb19 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.764092 19366 tablet_replica.cc:331] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19: stopping tablet replica
I20250831 13:57:52.764499 19366 raft_consensus.cc:2241] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:57:52.764848 19366 raft_consensus.cc:2270] T ae2e10d60e53469396554f7ff859aa48 P c5ef2f41fe5e49f3b889122d0efadb19 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.791828 19366 tablet_server.cc:195] TabletServer@127.18.233.131:0 shutdown complete.
I20250831 13:57:52.808720 19366 master.cc:561] Master@127.18.233.190:39183 shutting down...
I20250831 13:57:52.951197 19951 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20250831 13:57:52.952615 19366 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 1 LEADER]: Raft consensus shutting down.
I20250831 13:57:52.953153 19366 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 13:57:52.953490 19366 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7f907f12a2e34b21a5596a485e246e78: stopping tablet replica
I20250831 13:57:52.971954 19366 master.cc:583] Master@127.18.233.190:39183 shutdown complete.
0f6544748dcb4a1b88337dd4c55be644 4
998e50cfd9d94f17a96ecbf04b97496f 5
c5ef2f41fe5e49f3b889122d0efadb19 1
0f6544748dcb4a1b88337dd4c55be644 4
998e50cfd9d94f17a96ecbf04b97496f 3
c5ef2f41fe5e49f3b889122d0efadb19 3
0f6544748dcb4a1b88337dd4c55be644 1
998e50cfd9d94f17a96ecbf04b97496f 8
c5ef2f41fe5e49f3b889122d0efadb19 1
0f6544748dcb4a1b88337dd4c55be644 3
998e50cfd9d94f17a96ecbf04b97496f 4
c5ef2f41fe5e49f3b889122d0efadb19 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28675 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20250831 13:57:53.011447 19366 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.18.233.190:42663
I20250831 13:57:53.012429 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:57:53.017851 20375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:53.019455 20376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:53.019675 20378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:53.020488 19366 server_base.cc:1047] running on GCE node
I20250831 13:57:53.021358 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:57:53.021553 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:57:53.021660 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648673021647 us; error 0 us; skew 500 ppm
I20250831 13:57:53.022169 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:57:53.024464 19366 webserver.cc:480] Webserver started at http://127.18.233.190:38135/ using document root <none> and password file <none>
I20250831 13:57:53.024904 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:57:53.025077 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:57:53.025352 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:57:53.026472 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/master-0-root/instance:
uuid: "6f9e84d5d5f24202bf6812740505bbfd"
format_stamp: "Formatted at 2025-08-31 13:57:53 on dist-test-slave-sg9n"
I20250831 13:57:53.031013 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20250831 13:57:53.034128 20383 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.034855 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250831 13:57:53.035145 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/master-0-root
uuid: "6f9e84d5d5f24202bf6812740505bbfd"
format_stamp: "Formatted at 2025-08-31 13:57:53 on dist-test-slave-sg9n"
I20250831 13:57:53.035390 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:57:53.055588 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:57:53.056761 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:57:53.093181 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.190:42663
I20250831 13:57:53.093261 20434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.190:42663 every 8 connection(s)
I20250831 13:57:53.096918 20435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.106257 20435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd: Bootstrap starting.
I20250831 13:57:53.110540 20435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:53.114322 20435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd: No bootstrap required, opened a new log
I20250831 13:57:53.116560 20435 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9e84d5d5f24202bf6812740505bbfd" member_type: VOTER }
I20250831 13:57:53.116926 20435 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:53.117111 20435 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9e84d5d5f24202bf6812740505bbfd, State: Initialized, Role: FOLLOWER
I20250831 13:57:53.117545 20435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9e84d5d5f24202bf6812740505bbfd" member_type: VOTER }
I20250831 13:57:53.117925 20435 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250831 13:57:53.118104 20435 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250831 13:57:53.118319 20435 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:53.121922 20435 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9e84d5d5f24202bf6812740505bbfd" member_type: VOTER }
I20250831 13:57:53.122398 20435 leader_election.cc:304] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f9e84d5d5f24202bf6812740505bbfd; no voters:
I20250831 13:57:53.123399 20435 leader_election.cc:290] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250831 13:57:53.123637 20438 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:53.125097 20438 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [term 1 LEADER]: Becoming Leader. State: Replica: 6f9e84d5d5f24202bf6812740505bbfd, State: Running, Role: LEADER
I20250831 13:57:53.125759 20438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9e84d5d5f24202bf6812740505bbfd" member_type: VOTER }
I20250831 13:57:53.126398 20435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [sys.catalog]: configured and running, proceeding with master startup.
I20250831 13:57:53.130620 20440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f9e84d5d5f24202bf6812740505bbfd. Latest consensus state: current_term: 1 leader_uuid: "6f9e84d5d5f24202bf6812740505bbfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9e84d5d5f24202bf6812740505bbfd" member_type: VOTER } }
I20250831 13:57:53.130530 20439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f9e84d5d5f24202bf6812740505bbfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9e84d5d5f24202bf6812740505bbfd" member_type: VOTER } }
I20250831 13:57:53.131306 20440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [sys.catalog]: This master's current role is: LEADER
I20250831 13:57:53.131381 20439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd [sys.catalog]: This master's current role is: LEADER
I20250831 13:57:53.133283 20445 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250831 13:57:53.140674 20445 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250831 13:57:53.141626 19366 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250831 13:57:53.148607 20445 catalog_manager.cc:1349] Generated new cluster ID: d0a2b7e52b3c403f8305614b221d11b8
I20250831 13:57:53.148937 20445 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250831 13:57:53.167992 20445 catalog_manager.cc:1372] Generated new certificate authority record
I20250831 13:57:53.169263 20445 catalog_manager.cc:1506] Loading token signing keys...
I20250831 13:57:53.179113 20445 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd: Generated new TSK 0
I20250831 13:57:53.179626 20445 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250831 13:57:53.208510 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:57:53.215230 20457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:53.215113 20456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:53.217733 19366 server_base.cc:1047] running on GCE node
W20250831 13:57:53.217707 20459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:53.218851 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:57:53.219064 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:57:53.219228 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648673219207 us; error 0 us; skew 500 ppm
I20250831 13:57:53.219720 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:57:53.221797 19366 webserver.cc:480] Webserver started at http://127.18.233.129:43573/ using document root <none> and password file <none>
I20250831 13:57:53.222245 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:57:53.222419 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:57:53.222651 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:57:53.223749 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-0-root/instance:
uuid: "58aded50ab0d4609a053647eb11b1f48"
format_stamp: "Formatted at 2025-08-31 13:57:53 on dist-test-slave-sg9n"
I20250831 13:57:53.228288 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250831 13:57:53.231303 20464 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.232009 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250831 13:57:53.232271 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-0-root
uuid: "58aded50ab0d4609a053647eb11b1f48"
format_stamp: "Formatted at 2025-08-31 13:57:53 on dist-test-slave-sg9n"
I20250831 13:57:53.232506 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:57:53.247531 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:57:53.248721 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:57:53.250067 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:57:53.252137 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:57:53.252290 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.252560 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:57:53.252707 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.290923 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.129:35131
I20250831 13:57:53.290999 20526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.129:35131 every 8 connection(s)
I20250831 13:57:53.295436 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:57:53.302925 20531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:53.303704 20532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:53.305482 20534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:53.307251 19366 server_base.cc:1047] running on GCE node
I20250831 13:57:53.308712 20527 heartbeater.cc:344] Connected to a master server at 127.18.233.190:42663
I20250831 13:57:53.308713 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:57:53.309125 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:57:53.309168 20527 heartbeater.cc:461] Registering TS with master...
I20250831 13:57:53.309334 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648673309299 us; error 0 us; skew 500 ppm
I20250831 13:57:53.309903 20527 heartbeater.cc:507] Master 127.18.233.190:42663 requested a full tablet report, sending...
I20250831 13:57:53.310109 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:57:53.312045 20400 ts_manager.cc:194] Registered new tserver with Master: 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:57:53.312815 19366 webserver.cc:480] Webserver started at http://127.18.233.130:41203/ using document root <none> and password file <none>
I20250831 13:57:53.313458 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:57:53.313694 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:57:53.313994 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:57:53.314462 20400 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59924
I20250831 13:57:53.315510 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-1-root/instance:
uuid: "212b66976f874aa682cae0fb77928786"
format_stamp: "Formatted at 2025-08-31 13:57:53 on dist-test-slave-sg9n"
I20250831 13:57:53.319702 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250831 13:57:53.322644 20539 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.323331 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250831 13:57:53.323578 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-1-root
uuid: "212b66976f874aa682cae0fb77928786"
format_stamp: "Formatted at 2025-08-31 13:57:53 on dist-test-slave-sg9n"
I20250831 13:57:53.323824 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:57:53.337355 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:57:53.338445 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:57:53.339841 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:57:53.341962 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:57:53.342130 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.342376 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:57:53.342510 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.381256 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.130:44629
I20250831 13:57:53.381359 20601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.130:44629 every 8 connection(s)
I20250831 13:57:53.385658 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:57:53.392920 20605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:53.394490 20606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:57:53.397171 20608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:57:53.397763 20602 heartbeater.cc:344] Connected to a master server at 127.18.233.190:42663
I20250831 13:57:53.398185 20602 heartbeater.cc:461] Registering TS with master...
I20250831 13:57:53.398233 19366 server_base.cc:1047] running on GCE node
I20250831 13:57:53.399168 20602 heartbeater.cc:507] Master 127.18.233.190:42663 requested a full tablet report, sending...
I20250831 13:57:53.399302 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:57:53.399557 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:57:53.399760 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648673399740 us; error 0 us; skew 500 ppm
I20250831 13:57:53.400439 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:57:53.401549 20400 ts_manager.cc:194] Registered new tserver with Master: 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:53.403326 20400 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59926
I20250831 13:57:53.405531 19366 webserver.cc:480] Webserver started at http://127.18.233.131:43667/ using document root <none> and password file <none>
I20250831 13:57:53.406162 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:57:53.406394 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:57:53.406697 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:57:53.408004 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-2-root/instance:
uuid: "44d163094add49049cc749547d891602"
format_stamp: "Formatted at 2025-08-31 13:57:53 on dist-test-slave-sg9n"
I20250831 13:57:53.412091 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20250831 13:57:53.415066 20613 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.415874 19366 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250831 13:57:53.416162 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-2-root
uuid: "44d163094add49049cc749547d891602"
format_stamp: "Formatted at 2025-08-31 13:57:53 on dist-test-slave-sg9n"
I20250831 13:57:53.416397 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:57:53.437004 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:57:53.438102 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:57:53.439489 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:57:53.441596 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:57:53.441784 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.442016 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:57:53.442176 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:57:53.483404 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.131:45067
I20250831 13:57:53.483486 20675 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.131:45067 every 8 connection(s)
I20250831 13:57:53.498337 20676 heartbeater.cc:344] Connected to a master server at 127.18.233.190:42663
I20250831 13:57:53.498814 20676 heartbeater.cc:461] Registering TS with master...
I20250831 13:57:53.499558 20676 heartbeater.cc:507] Master 127.18.233.190:42663 requested a full tablet report, sending...
I20250831 13:57:53.501327 20400 ts_manager.cc:194] Registered new tserver with Master: 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:53.501819 19366 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015403037s
I20250831 13:57:53.502969 20400 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59942
I20250831 13:57:53.504586 19366 test_util.cc:276] Using random seed: -1540494024
I20250831 13:57:53.526002 20400 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59948:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250831 13:57:53.532032 20400 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250831 13:57:53.721395 20640 tablet_service.cc:1468] Processing CreateTablet for tablet 099d69058d8b46999f57c3142f80acd0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250831 13:57:53.722198 20639 tablet_service.cc:1468] Processing CreateTablet for tablet 24ba1ebe867e4fbb9b5f869fbfd4b489 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250831 13:57:53.722939 20640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099d69058d8b46999f57c3142f80acd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.724417 20638 tablet_service.cc:1468] Processing CreateTablet for tablet ca1477952e8c40adb14c71db5a7a4b92 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250831 13:57:53.726855 20641 tablet_service.cc:1468] Processing CreateTablet for tablet 62d7feaf5302450caacdf561ce6f06d8 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250831 13:57:53.726850 20637 tablet_service.cc:1468] Processing CreateTablet for tablet 6d512286b29841fbaba3df87e23ceff4 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250831 13:57:53.728025 20565 tablet_service.cc:1468] Processing CreateTablet for tablet 24ba1ebe867e4fbb9b5f869fbfd4b489 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250831 13:57:53.729298 20565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ba1ebe867e4fbb9b5f869fbfd4b489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.730224 20636 tablet_service.cc:1468] Processing CreateTablet for tablet 712d44ed33ec4e37b2bef30d18b3495b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250831 13:57:53.731745 20560 tablet_service.cc:1468] Processing CreateTablet for tablet 0e8142901f0248e7b0565a5be0dbda2b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250831 13:57:53.732425 20635 tablet_service.cc:1468] Processing CreateTablet for tablet 192560b5dd5040279d722ef31a62c940 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250831 13:57:53.732955 20560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8142901f0248e7b0565a5be0dbda2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.733824 20638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1477952e8c40adb14c71db5a7a4b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.734472 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d7feaf5302450caacdf561ce6f06d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.735168 20637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d512286b29841fbaba3df87e23ceff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.736315 20639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ba1ebe867e4fbb9b5f869fbfd4b489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.737023 20567 tablet_service.cc:1468] Processing CreateTablet for tablet 62d7feaf5302450caacdf561ce6f06d8 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250831 13:57:53.738204 20567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d7feaf5302450caacdf561ce6f06d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.739068 20635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192560b5dd5040279d722ef31a62c940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.739758 20564 tablet_service.cc:1468] Processing CreateTablet for tablet ca1477952e8c40adb14c71db5a7a4b92 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250831 13:57:53.740921 20564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1477952e8c40adb14c71db5a7a4b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.743036 20636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712d44ed33ec4e37b2bef30d18b3495b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.743773 20566 tablet_service.cc:1468] Processing CreateTablet for tablet 099d69058d8b46999f57c3142f80acd0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250831 13:57:53.744985 20566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099d69058d8b46999f57c3142f80acd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.747783 20562 tablet_service.cc:1468] Processing CreateTablet for tablet 712d44ed33ec4e37b2bef30d18b3495b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250831 13:57:53.748996 20562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712d44ed33ec4e37b2bef30d18b3495b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.749487 20634 tablet_service.cc:1468] Processing CreateTablet for tablet 0e8142901f0248e7b0565a5be0dbda2b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250831 13:57:53.750824 20634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8142901f0248e7b0565a5be0dbda2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.751797 20558 tablet_service.cc:1468] Processing CreateTablet for tablet 7562a669789845418660f6f1c3109b1e (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250831 13:57:53.752959 20558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7562a669789845418660f6f1c3109b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.755961 20563 tablet_service.cc:1468] Processing CreateTablet for tablet 6d512286b29841fbaba3df87e23ceff4 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250831 13:57:53.757187 20563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d512286b29841fbaba3df87e23ceff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.757545 20559 tablet_service.cc:1468] Processing CreateTablet for tablet 3359826124a94225a4cd8932ecb7fcff (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250831 13:57:53.758858 20559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3359826124a94225a4cd8932ecb7fcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.726884 20561 tablet_service.cc:1468] Processing CreateTablet for tablet 192560b5dd5040279d722ef31a62c940 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250831 13:57:53.764030 20561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192560b5dd5040279d722ef31a62c940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.772272 20633 tablet_service.cc:1468] Processing CreateTablet for tablet 3359826124a94225a4cd8932ecb7fcff (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250831 13:57:53.775408 20632 tablet_service.cc:1468] Processing CreateTablet for tablet 7562a669789845418660f6f1c3109b1e (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250831 13:57:53.779007 20633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3359826124a94225a4cd8932ecb7fcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.779776 20632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7562a669789845418660f6f1c3109b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.824687 20492 tablet_service.cc:1468] Processing CreateTablet for tablet 62d7feaf5302450caacdf561ce6f06d8 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION VALUES < 36398027
I20250831 13:57:53.826038 20492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62d7feaf5302450caacdf561ce6f06d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.827389 20484 tablet_service.cc:1468] Processing CreateTablet for tablet 3359826124a94225a4cd8932ecb7fcff (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20250831 13:57:53.827852 20491 tablet_service.cc:1468] Processing CreateTablet for tablet 099d69058d8b46999f57c3142f80acd0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20250831 13:57:53.828660 20484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3359826124a94225a4cd8932ecb7fcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.830024 20490 tablet_service.cc:1468] Processing CreateTablet for tablet 24ba1ebe867e4fbb9b5f869fbfd4b489 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20250831 13:57:53.831314 20490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ba1ebe867e4fbb9b5f869fbfd4b489. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.832315 20483 tablet_service.cc:1468] Processing CreateTablet for tablet 7562a669789845418660f6f1c3109b1e (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20250831 13:57:53.833648 20483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7562a669789845418660f6f1c3109b1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.837145 20488 tablet_service.cc:1468] Processing CreateTablet for tablet 6d512286b29841fbaba3df87e23ceff4 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20250831 13:57:53.839946 20485 tablet_service.cc:1468] Processing CreateTablet for tablet 0e8142901f0248e7b0565a5be0dbda2b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20250831 13:57:53.835835 20489 tablet_service.cc:1468] Processing CreateTablet for tablet ca1477952e8c40adb14c71db5a7a4b92 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20250831 13:57:53.840073 20486 tablet_service.cc:1468] Processing CreateTablet for tablet 192560b5dd5040279d722ef31a62c940 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20250831 13:57:53.841632 20487 tablet_service.cc:1468] Processing CreateTablet for tablet 712d44ed33ec4e37b2bef30d18b3495b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20250831 13:57:53.844427 20491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099d69058d8b46999f57c3142f80acd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.847013 20487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 712d44ed33ec4e37b2bef30d18b3495b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.850971 20488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d512286b29841fbaba3df87e23ceff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.853436 20489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1477952e8c40adb14c71db5a7a4b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.854974 20485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8142901f0248e7b0565a5be0dbda2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.857519 20486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 192560b5dd5040279d722ef31a62c940. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.858651 20640 tablet_service.cc:1468] Processing CreateTablet for tablet 9bc5bbba290e48548aa3237b4d8e1bd1 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250831 13:57:53.867375 20640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc5bbba290e48548aa3237b4d8e1bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.867247 20641 tablet_service.cc:1468] Processing CreateTablet for tablet b2d4d2e85a054797ae54ac949008c1f1 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250831 13:57:53.877128 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d4d2e85a054797ae54ac949008c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.876964 20634 tablet_service.cc:1468] Processing CreateTablet for tablet 6dd87c9794194ce3a9cb6e3b933c3215 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250831 13:57:53.869630 20565 tablet_service.cc:1468] Processing CreateTablet for tablet b2d4d2e85a054797ae54ac949008c1f1 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250831 13:57:53.879407 20565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d4d2e85a054797ae54ac949008c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.882192 20567 tablet_service.cc:1468] Processing CreateTablet for tablet 9bc5bbba290e48548aa3237b4d8e1bd1 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250831 13:57:53.883841 20567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc5bbba290e48548aa3237b4d8e1bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.878561 20634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd87c9794194ce3a9cb6e3b933c3215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.888784 20559 tablet_service.cc:1468] Processing CreateTablet for tablet 6dd87c9794194ce3a9cb6e3b933c3215 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250831 13:57:53.890030 20559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd87c9794194ce3a9cb6e3b933c3215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.904526 20635 tablet_service.cc:1468] Processing CreateTablet for tablet ccee615469564d43800cb6f12f325e22 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250831 13:57:53.906838 20691 tablet_bootstrap.cc:492] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:53.910950 20635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccee615469564d43800cb6f12f325e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.911193 20633 tablet_service.cc:1468] Processing CreateTablet for tablet c8699373860740a9b912a1ce516e1510 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250831 13:57:53.912766 20633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8699373860740a9b912a1ce516e1510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.924927 20566 tablet_service.cc:1468] Processing CreateTablet for tablet ccee615469564d43800cb6f12f325e22 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250831 13:57:53.928239 20561 tablet_service.cc:1468] Processing CreateTablet for tablet 0c14599239fd4c99b3331692a513a092 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250831 13:57:53.939355 20692 tablet_bootstrap.cc:492] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:53.945138 20692 tablet_bootstrap.cc:654] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:53.946946 20559 tablet_service.cc:1468] Processing CreateTablet for tablet 00c91580fd1b4c45a280862eedb8e1ac (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250831 13:57:53.947531 20563 tablet_service.cc:1468] Processing CreateTablet for tablet c719f2a4c3ea4418937337e8e3dd2164 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250831 13:57:53.953469 20693 tablet_bootstrap.cc:492] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:53.953809 20490 tablet_service.cc:1468] Processing CreateTablet for tablet 9bc5bbba290e48548aa3237b4d8e1bd1 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20250831 13:57:53.955142 20490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bc5bbba290e48548aa3237b4d8e1bd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.964031 20563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c719f2a4c3ea4418937337e8e3dd2164. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.964447 20483 tablet_service.cc:1468] Processing CreateTablet for tablet 6dd87c9794194ce3a9cb6e3b933c3215 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20250831 13:57:53.964923 20562 tablet_service.cc:1468] Processing CreateTablet for tablet 5806109eefc8457d8e9110b9ec030873 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250831 13:57:53.966246 20483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd87c9794194ce3a9cb6e3b933c3215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.966246 20562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5806109eefc8457d8e9110b9ec030873. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.969460 20566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccee615469564d43800cb6f12f325e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.971297 20559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c91580fd1b4c45a280862eedb8e1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.979297 20632 tablet_service.cc:1468] Processing CreateTablet for tablet 0c14599239fd4c99b3331692a513a092 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250831 13:57:53.939361 20485 tablet_service.cc:1468] Processing CreateTablet for tablet b2d4d2e85a054797ae54ac949008c1f1 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20250831 13:57:53.992266 20485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d4d2e85a054797ae54ac949008c1f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:53.996795 20691 tablet_bootstrap.cc:654] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.004307 20486 tablet_service.cc:1468] Processing CreateTablet for tablet ccee615469564d43800cb6f12f325e22 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20250831 13:57:54.005558 20486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccee615469564d43800cb6f12f325e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.016894 20490 tablet_service.cc:1468] Processing CreateTablet for tablet c8699373860740a9b912a1ce516e1510 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250831 13:57:54.027689 20563 tablet_service.cc:1468] Processing CreateTablet for tablet da27681405bf47e28acb6b95258c0b29 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250831 13:57:54.028963 20563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da27681405bf47e28acb6b95258c0b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.031370 20693 tablet_bootstrap.cc:654] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.039690 20561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c14599239fd4c99b3331692a513a092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.044698 20491 tablet_service.cc:1468] Processing CreateTablet for tablet 0c14599239fd4c99b3331692a513a092 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20250831 13:57:54.045029 20487 tablet_service.cc:1468] Processing CreateTablet for tablet 00c91580fd1b4c45a280862eedb8e1ac (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250831 13:57:54.045825 20490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8699373860740a9b912a1ce516e1510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.046257 20564 tablet_service.cc:1468] Processing CreateTablet for tablet c8699373860740a9b912a1ce516e1510 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20250831 13:57:54.047426 20564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8699373860740a9b912a1ce516e1510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.050729 20487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c91580fd1b4c45a280862eedb8e1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.056849 20639 tablet_service.cc:1468] Processing CreateTablet for tablet c719f2a4c3ea4418937337e8e3dd2164 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250831 13:57:54.058022 20639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c719f2a4c3ea4418937337e8e3dd2164. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.062093 20638 tablet_service.cc:1468] Processing CreateTablet for tablet 5806109eefc8457d8e9110b9ec030873 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250831 13:57:54.064309 20638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5806109eefc8457d8e9110b9ec030873. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.066584 20632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c14599239fd4c99b3331692a513a092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.068763 20492 tablet_service.cc:1468] Processing CreateTablet for tablet c719f2a4c3ea4418937337e8e3dd2164 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20250831 13:57:54.069948 20492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c719f2a4c3ea4418937337e8e3dd2164. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.072034 20488 tablet_service.cc:1468] Processing CreateTablet for tablet 5806109eefc8457d8e9110b9ec030873 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20250831 13:57:54.073194 20488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5806109eefc8457d8e9110b9ec030873. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.073101 20484 tablet_service.cc:1468] Processing CreateTablet for tablet da27681405bf47e28acb6b95258c0b29 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250831 13:57:54.074324 20484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da27681405bf47e28acb6b95258c0b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.056849 20637 tablet_service.cc:1468] Processing CreateTablet for tablet 00c91580fd1b4c45a280862eedb8e1ac (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20250831 13:57:54.078712 20637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c91580fd1b4c45a280862eedb8e1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.081921 20491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c14599239fd4c99b3331692a513a092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.088819 20636 tablet_service.cc:1468] Processing CreateTablet for tablet da27681405bf47e28acb6b95258c0b29 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20250831 13:57:54.090049 20636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da27681405bf47e28acb6b95258c0b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.095454 20489 tablet_service.cc:1468] Processing CreateTablet for tablet e7166435c35c41f580e6cfbdaa2d0c39 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250831 13:57:54.096807 20489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7166435c35c41f580e6cfbdaa2d0c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.103967 20567 tablet_service.cc:1468] Processing CreateTablet for tablet e7166435c35c41f580e6cfbdaa2d0c39 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250831 13:57:54.105219 20567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7166435c35c41f580e6cfbdaa2d0c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.109172 20636 tablet_service.cc:1468] Processing CreateTablet for tablet e7166435c35c41f580e6cfbdaa2d0c39 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20250831 13:57:54.111907 20636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7166435c35c41f580e6cfbdaa2d0c39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.112510 20640 tablet_service.cc:1468] Processing CreateTablet for tablet c933fb1eb5c54d0b8b0a6cc59ba2aef5 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250831 13:57:54.113685 20640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c933fb1eb5c54d0b8b0a6cc59ba2aef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.109668 20562 tablet_service.cc:1468] Processing CreateTablet for tablet c933fb1eb5c54d0b8b0a6cc59ba2aef5 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250831 13:57:54.110694 20491 tablet_service.cc:1468] Processing CreateTablet for tablet c933fb1eb5c54d0b8b0a6cc59ba2aef5 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20250831 13:57:54.115911 20491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c933fb1eb5c54d0b8b0a6cc59ba2aef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.115952 20562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c933fb1eb5c54d0b8b0a6cc59ba2aef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.132110 20559 tablet_service.cc:1468] Processing CreateTablet for tablet 1c85d8618b084c51b4e509df8650069d (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250831 13:57:54.132550 20490 tablet_service.cc:1468] Processing CreateTablet for tablet 1c85d8618b084c51b4e509df8650069d (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250831 13:57:54.133451 20559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c85d8618b084c51b4e509df8650069d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 13:57:54.136546 20448 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:57:54.145123 20490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c85d8618b084c51b4e509df8650069d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.156988 20692 tablet_bootstrap.cc:492] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:54.157511 20692 ts_tablet_manager.cc:1397] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.218s user 0.033s sys 0.064s
I20250831 13:57:54.160199 20692 raft_consensus.cc:357] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.160930 20692 raft_consensus.cc:383] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.161233 20692 raft_consensus.cc:738] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.164170 20693 tablet_bootstrap.cc:492] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:54.164631 20693 ts_tablet_manager.cc:1397] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.211s user 0.013s sys 0.000s
I20250831 13:57:54.164170 20691 tablet_bootstrap.cc:492] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:54.167007 20691 ts_tablet_manager.cc:1397] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.261s user 0.024s sys 0.039s
I20250831 13:57:54.169811 20691 raft_consensus.cc:357] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.180708 20636 tablet_service.cc:1468] Processing CreateTablet for tablet 1c85d8618b084c51b4e509df8650069d (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20250831 13:57:54.181977 20636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c85d8618b084c51b4e509df8650069d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.170486 20691 raft_consensus.cc:383] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.182835 20691 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.183645 20691 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.187317 20641 tablet_service.cc:1468] Processing CreateTablet for tablet 7f52ce7625424dcc943a573347b456bb (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250831 13:57:54.187711 20490 tablet_service.cc:1468] Processing CreateTablet for tablet 501e1c372209405c80034fe2411aaa22 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250831 13:57:54.188621 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f52ce7625424dcc943a573347b456bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.188455 20567 tablet_service.cc:1468] Processing CreateTablet for tablet 501e1c372209405c80034fe2411aaa22 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250831 13:57:54.188957 20490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501e1c372209405c80034fe2411aaa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.189682 20567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501e1c372209405c80034fe2411aaa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.185710 20492 tablet_service.cc:1468] Processing CreateTablet for tablet 7f52ce7625424dcc943a573347b456bb (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250831 13:57:54.197234 20633 tablet_service.cc:1468] Processing CreateTablet for tablet ad68e3a13b9e479791179f35e91bea4b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250831 13:57:54.198949 20638 tablet_service.cc:1468] Processing CreateTablet for tablet 501e1c372209405c80034fe2411aaa22 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20250831 13:57:54.203811 20564 tablet_service.cc:1468] Processing CreateTablet for tablet ad68e3a13b9e479791179f35e91bea4b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250831 13:57:54.205001 20564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad68e3a13b9e479791179f35e91bea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.206828 20693 raft_consensus.cc:357] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.207551 20693 raft_consensus.cc:383] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.207846 20693 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.207801 20486 tablet_service.cc:1468] Processing CreateTablet for tablet bf7dfe5303d444eeb5f33f78d248eaec (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250831 13:57:54.208766 20693 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.209934 20692 consensus_queue.cc:260] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.211454 20565 tablet_service.cc:1468] Processing CreateTablet for tablet 7f52ce7625424dcc943a573347b456bb (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20250831 13:57:54.212184 20485 tablet_service.cc:1468] Processing CreateTablet for tablet ad68e3a13b9e479791179f35e91bea4b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20250831 13:57:54.212630 20565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f52ce7625424dcc943a573347b456bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.194183 20492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f52ce7625424dcc943a573347b456bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.231720 20486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7dfe5303d444eeb5f33f78d248eaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.233002 20563 tablet_service.cc:1468] Processing CreateTablet for tablet bf7dfe5303d444eeb5f33f78d248eaec (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250831 13:57:54.234236 20563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7dfe5303d444eeb5f33f78d248eaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.235091 20692 ts_tablet_manager.cc:1428] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.077s user 0.008s sys 0.003s
I20250831 13:57:54.235289 20676 heartbeater.cc:499] Master 127.18.233.190:42663 was elected leader, sending a full tablet report...
I20250831 13:57:54.235808 20602 heartbeater.cc:499] Master 127.18.233.190:42663 was elected leader, sending a full tablet report...
I20250831 13:57:54.242370 20692 tablet_bootstrap.cc:492] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:54.243964 20693 ts_tablet_manager.cc:1428] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.079s user 0.008s sys 0.000s
I20250831 13:57:54.244794 20485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad68e3a13b9e479791179f35e91bea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.245434 20527 heartbeater.cc:499] Master 127.18.233.190:42663 was elected leader, sending a full tablet report...
I20250831 13:57:54.245725 20693 tablet_bootstrap.cc:492] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:54.256625 20633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad68e3a13b9e479791179f35e91bea4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.257737 20488 tablet_service.cc:1468] Processing CreateTablet for tablet ca82b3cf6a194875b3a6e31278e37e51 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250831 13:57:54.259043 20488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca82b3cf6a194875b3a6e31278e37e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.261781 20693 tablet_bootstrap.cc:654] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.263057 20638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 501e1c372209405c80034fe2411aaa22. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.271843 20691 ts_tablet_manager.cc:1428] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.105s user 0.009s sys 0.011s
I20250831 13:57:54.271776 20641 tablet_service.cc:1468] Processing CreateTablet for tablet bf7dfe5303d444eeb5f33f78d248eaec (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20250831 13:57:54.272925 20691 tablet_bootstrap.cc:492] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:54.275172 20698 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:54.275611 20698 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.278199 20698 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:54.278931 20632 tablet_service.cc:1468] Processing CreateTablet for tablet ca82b3cf6a194875b3a6e31278e37e51 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250831 13:57:54.279635 20562 tablet_service.cc:1468] Processing CreateTablet for tablet ca82b3cf6a194875b3a6e31278e37e51 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20250831 13:57:54.281042 20562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca82b3cf6a194875b3a6e31278e37e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.282253 20484 tablet_service.cc:1468] Processing CreateTablet for tablet bb59414ade0543ef8c7f047104f39576 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250831 13:57:54.273056 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf7dfe5303d444eeb5f33f78d248eaec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.283566 20484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb59414ade0543ef8c7f047104f39576. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.291159 20565 tablet_service.cc:1468] Processing CreateTablet for tablet bb59414ade0543ef8c7f047104f39576 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250831 13:57:54.292624 20565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb59414ade0543ef8c7f047104f39576. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 13:57:54.295331 20528 tablet.cc:2378] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250831 13:57:54.306655 20692 tablet_bootstrap.cc:654] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.319097 20487 tablet_service.cc:1468] Processing CreateTablet for tablet f10d72d324374615bbfb06fd2c0e548d (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250831 13:57:54.331051 20632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca82b3cf6a194875b3a6e31278e37e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.331508 20487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10d72d324374615bbfb06fd2c0e548d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.334074 20691 tablet_bootstrap.cc:654] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.338853 20691 tablet_bootstrap.cc:492] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:54.319653 20693 tablet_bootstrap.cc:492] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:54.340623 20691 ts_tablet_manager.cc:1397] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.002s
I20250831 13:57:54.319108 20637 tablet_service.cc:1468] Processing CreateTablet for tablet bb59414ade0543ef8c7f047104f39576 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20250831 13:57:54.340813 20693 ts_tablet_manager.cc:1397] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.095s user 0.014s sys 0.002s
I20250831 13:57:54.341719 20637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb59414ade0543ef8c7f047104f39576. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.343305 20691 raft_consensus.cc:357] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:54.344045 20691 raft_consensus.cc:383] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.344341 20691 raft_consensus.cc:738] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.345134 20691 consensus_queue.cc:260] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:54.347589 20691 ts_tablet_manager.cc:1428] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 13:57:54.348441 20691 tablet_bootstrap.cc:492] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:54.350893 20693 raft_consensus.cc:357] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.354300 20691 tablet_bootstrap.cc:654] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.362847 20693 raft_consensus.cc:383] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.363188 20693 raft_consensus.cc:738] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.363900 20693 consensus_queue.cc:260] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.365876 20693 ts_tablet_manager.cc:1428] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.024s user 0.000s sys 0.003s
I20250831 13:57:54.366569 20693 tablet_bootstrap.cc:492] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:54.369109 20692 tablet_bootstrap.cc:492] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:54.369575 20692 ts_tablet_manager.cc:1397] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.127s user 0.013s sys 0.001s
I20250831 13:57:54.372359 20692 raft_consensus.cc:357] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.373077 20692 raft_consensus.cc:383] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.373368 20692 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.373574 20693 tablet_bootstrap.cc:654] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.374110 20692 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.376084 20691 tablet_bootstrap.cc:492] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:54.414940 20492 tablet_service.cc:1468] Processing CreateTablet for tablet 512ab0e0a72d4872b147e21acc0832fd (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250831 13:57:54.417536 20492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512ab0e0a72d4872b147e21acc0832fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.417788 20559 tablet_service.cc:1468] Processing CreateTablet for tablet 90a0a56228d04660b3cb351485be6593 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250831 13:57:54.419751 20559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a0a56228d04660b3cb351485be6593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.419987 20564 tablet_service.cc:1468] Processing CreateTablet for tablet 512ab0e0a72d4872b147e21acc0832fd (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250831 13:57:54.420142 20563 tablet_service.cc:1468] Processing CreateTablet for tablet f10d72d324374615bbfb06fd2c0e548d (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250831 13:57:54.421693 20564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512ab0e0a72d4872b147e21acc0832fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.423305 20563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10d72d324374615bbfb06fd2c0e548d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.423185 20486 tablet_service.cc:1468] Processing CreateTablet for tablet 90a0a56228d04660b3cb351485be6593 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250831 13:57:54.424542 20486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a0a56228d04660b3cb351485be6593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.431911 20561 tablet_service.cc:1468] Processing CreateTablet for tablet bca7755948d54e9ab6dbefcbf93c927f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250831 13:57:54.433257 20561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bca7755948d54e9ab6dbefcbf93c927f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.435730 20491 tablet_service.cc:1468] Processing CreateTablet for tablet 8b2d32573f02435681e911c3badba77b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250831 13:57:54.438442 20483 tablet_service.cc:1468] Processing CreateTablet for tablet bca7755948d54e9ab6dbefcbf93c927f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250831 13:57:54.439807 20491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2d32573f02435681e911c3badba77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.443022 20483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bca7755948d54e9ab6dbefcbf93c927f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.442771 20560 tablet_service.cc:1468] Processing CreateTablet for tablet 795afdca08334d5aa9a9b944be40f34f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250831 13:57:54.444103 20560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795afdca08334d5aa9a9b944be40f34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.443966 20566 tablet_service.cc:1468] Processing CreateTablet for tablet 8b2d32573f02435681e911c3badba77b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250831 13:57:54.445281 20566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2d32573f02435681e911c3badba77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.446188 20567 tablet_service.cc:1468] Processing CreateTablet for tablet 223426ea4b53470699bceb4fb06dd992 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250831 13:57:54.447428 20567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223426ea4b53470699bceb4fb06dd992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.452306 20489 tablet_service.cc:1468] Processing CreateTablet for tablet 795afdca08334d5aa9a9b944be40f34f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250831 13:57:54.453608 20489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795afdca08334d5aa9a9b944be40f34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.455519 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:54.457158 20577 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:54.421634 20490 tablet_service.cc:1468] Processing CreateTablet for tablet 223426ea4b53470699bceb4fb06dd992 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250831 13:57:54.459239 20490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223426ea4b53470699bceb4fb06dd992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.459232 20467 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:54.468456 20632 tablet_service.cc:1468] Processing CreateTablet for tablet 512ab0e0a72d4872b147e21acc0832fd (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20250831 13:57:54.471907 20632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512ab0e0a72d4872b147e21acc0832fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.472033 20638 tablet_service.cc:1468] Processing CreateTablet for tablet 90a0a56228d04660b3cb351485be6593 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20250831 13:57:54.473259 20638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a0a56228d04660b3cb351485be6593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.476114 20633 tablet_service.cc:1468] Processing CreateTablet for tablet 223426ea4b53470699bceb4fb06dd992 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20250831 13:57:54.477432 20633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223426ea4b53470699bceb4fb06dd992. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.479969 20641 tablet_service.cc:1468] Processing CreateTablet for tablet f10d72d324374615bbfb06fd2c0e548d (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20250831 13:57:54.480337 20636 tablet_service.cc:1468] Processing CreateTablet for tablet bca7755948d54e9ab6dbefcbf93c927f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20250831 13:57:54.481204 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f10d72d324374615bbfb06fd2c0e548d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.483845 20639 tablet_service.cc:1468] Processing CreateTablet for tablet 8b2d32573f02435681e911c3badba77b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20250831 13:57:54.485095 20639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b2d32573f02435681e911c3badba77b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.487536 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:54.488473 20640 tablet_service.cc:1468] Processing CreateTablet for tablet 795afdca08334d5aa9a9b944be40f34f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20250831 13:57:54.489912 20640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 795afdca08334d5aa9a9b944be40f34f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.493382 20635 tablet_service.cc:1468] Processing CreateTablet for tablet 9970f4ff4e4b4d8b8beceaf00d20c3f6 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250831 13:57:54.496426 20634 tablet_service.cc:1468] Processing CreateTablet for tablet 995d9ef0c84a4aae8581ab92c0607fdf (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250831 13:57:54.498051 20635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9970f4ff4e4b4d8b8beceaf00d20c3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.499857 20634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995d9ef0c84a4aae8581ab92c0607fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.501488 20636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bca7755948d54e9ab6dbefcbf93c927f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.502622 20691 ts_tablet_manager.cc:1397] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.154s user 0.016s sys 0.003s
I20250831 13:57:54.507058 20691 raft_consensus.cc:357] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.507957 20691 raft_consensus.cc:383] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.508262 20691 raft_consensus.cc:738] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.509106 20691 consensus_queue.cc:260] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.519057 20691 ts_tablet_manager.cc:1428] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.016s user 0.005s sys 0.004s
I20250831 13:57:54.519227 20485 tablet_service.cc:1468] Processing CreateTablet for tablet 9970f4ff4e4b4d8b8beceaf00d20c3f6 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250831 13:57:54.520278 20691 tablet_bootstrap.cc:492] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:54.520551 20485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9970f4ff4e4b4d8b8beceaf00d20c3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.532397 20565 tablet_service.cc:1468] Processing CreateTablet for tablet 9970f4ff4e4b4d8b8beceaf00d20c3f6 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20250831 13:57:54.533689 20565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9970f4ff4e4b4d8b8beceaf00d20c3f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.535892 20558 tablet_service.cc:1468] Processing CreateTablet for tablet 995d9ef0c84a4aae8581ab92c0607fdf (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250831 13:57:54.537236 20558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995d9ef0c84a4aae8581ab92c0607fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.539853 20691 tablet_bootstrap.cc:654] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.563547 20561 tablet_service.cc:1468] Processing CreateTablet for tablet 2c47fb589f98457ab81d63e6b9d19fd0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250831 13:57:54.567031 20700 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:54.555878 20484 tablet_service.cc:1468] Processing CreateTablet for tablet 995d9ef0c84a4aae8581ab92c0607fdf (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20250831 13:57:54.556680 20487 tablet_service.cc:1468] Processing CreateTablet for tablet 2c47fb589f98457ab81d63e6b9d19fd0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250831 13:57:54.567485 20700 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:54.567874 20700 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:54.568221 20487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c47fb589f98457ab81d63e6b9d19fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.571208 20699 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:54.571652 20699 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.559507 20560 tablet_service.cc:1468] Processing CreateTablet for tablet 79dc07c2112a46fe82d7250aa95bd6dc (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250831 13:57:54.553762 20633 tablet_service.cc:1468] Processing CreateTablet for tablet 2c47fb589f98457ab81d63e6b9d19fd0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20250831 13:57:54.376412 20692 ts_tablet_manager.cc:1428] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 13:57:54.581979 20692 tablet_bootstrap.cc:492] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:54.582073 20699 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:54.574463 20484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 995d9ef0c84a4aae8581ab92c0607fdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.576536 20691 tablet_bootstrap.cc:492] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:54.576753 20562 tablet_service.cc:1468] Processing CreateTablet for tablet 319db728a12e43e5808e70ff91e5155f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250831 13:57:54.579978 20637 tablet_service.cc:1468] Processing CreateTablet for tablet 79dc07c2112a46fe82d7250aa95bd6dc (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250831 13:57:54.592202 20637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79dc07c2112a46fe82d7250aa95bd6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.597079 20491 tablet_service.cc:1468] Processing CreateTablet for tablet 79dc07c2112a46fe82d7250aa95bd6dc (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20250831 13:57:54.599093 20691 ts_tablet_manager.cc:1397] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.008s
I20250831 13:57:54.604277 20693 tablet_bootstrap.cc:492] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:54.604172 20486 tablet_service.cc:1468] Processing CreateTablet for tablet a7956988b57844cfad1028024ac745e6 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250831 13:57:54.604858 20693 ts_tablet_manager.cc:1397] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.238s user 0.004s sys 0.006s
I20250831 13:57:54.612424 20691 raft_consensus.cc:357] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.613126 20691 raft_consensus.cc:383] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.613395 20691 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.613540 20693 raft_consensus.cc:357] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.614209 20693 raft_consensus.cc:383] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:54.605540 20565 tablet_service.cc:1468] Processing CreateTablet for tablet 1e3ed7784853456bbe218b149399b557 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250831 13:57:54.614498 20693 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:54.615726 20693 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.616470 20567 tablet_service.cc:1468] Processing CreateTablet for tablet a7956988b57844cfad1028024ac745e6 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20250831 13:57:54.618350 20693 ts_tablet_manager.cc:1428] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:57:54.619480 20693 tablet_bootstrap.cc:492] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:54.617498 20691 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.628285 20491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79dc07c2112a46fe82d7250aa95bd6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.628198 20639 tablet_service.cc:1468] Processing CreateTablet for tablet 319db728a12e43e5808e70ff91e5155f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250831 13:57:54.628492 20697 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:54.629436 20639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319db728a12e43e5808e70ff91e5155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.629006 20697 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.590035 20561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c47fb589f98457ab81d63e6b9d19fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.633176 20697 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:54.644546 20638 tablet_service.cc:1468] Processing CreateTablet for tablet a7956988b57844cfad1028024ac745e6 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
W20250831 13:57:54.646862 20466 leader_election.cc:343] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.647879 20566 tablet_service.cc:1468] Processing CreateTablet for tablet f29595dd2c044618bd6cb4517fbc5c7b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250831 13:57:54.650494 20558 tablet_service.cc:1468] Processing CreateTablet for tablet 8de1861ba5b5442381501e5b214a51f9 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250831 13:57:54.643635 20691 ts_tablet_manager.cc:1428] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.043s user 0.002s sys 0.005s
I20250831 13:57:54.664129 20641 tablet_service.cc:1468] Processing CreateTablet for tablet 1e3ed7784853456bbe218b149399b557 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250831 13:57:54.614601 20700 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.676680 20700 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:54.677317 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:54.677866 20577 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:54.655509 20486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7956988b57844cfad1028024ac745e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.682567 20489 tablet_service.cc:1468] Processing CreateTablet for tablet 319db728a12e43e5808e70ff91e5155f (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20250831 13:57:54.686851 20558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de1861ba5b5442381501e5b214a51f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.688892 20563 tablet_service.cc:1468] Processing CreateTablet for tablet 21eaf12e87504154aa74e46187b415ae (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250831 13:57:54.670241 20559 tablet_service.cc:1468] Processing CreateTablet for tablet b9f875f43094448cbafb11704a281c08 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250831 13:57:54.692497 20485 tablet_service.cc:1468] Processing CreateTablet for tablet e93846478efb47a1b008fc73d5d25914 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250831 13:57:54.662642 20633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c47fb589f98457ab81d63e6b9d19fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.701372 20697 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:54.697597 20492 tablet_service.cc:1468] Processing CreateTablet for tablet f29595dd2c044618bd6cb4517fbc5c7b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250831 13:57:54.712616 20565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3ed7784853456bbe218b149399b557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.652400 20488 tablet_service.cc:1468] Processing CreateTablet for tablet 1e3ed7784853456bbe218b149399b557 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20250831 13:57:54.718886 20638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7956988b57844cfad1028024ac745e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.719904 20490 tablet_service.cc:1468] Processing CreateTablet for tablet 21eaf12e87504154aa74e46187b415ae (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250831 13:57:54.713670 20697 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:54.732707 20697 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:57:54.733111 20697 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:54.733491 20697 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.709482 20564 tablet_service.cc:1468] Processing CreateTablet for tablet e93846478efb47a1b008fc73d5d25914 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250831 13:57:54.723174 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3ed7784853456bbe218b149399b557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.727710 20559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9f875f43094448cbafb11704a281c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.723204 20691 tablet_bootstrap.cc:492] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:54.728258 20693 tablet_bootstrap.cc:654] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.712858 20483 tablet_service.cc:1468] Processing CreateTablet for tablet b9f875f43094448cbafb11704a281c08 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250831 13:57:54.743566 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:54.747007 20566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29595dd2c044618bd6cb4517fbc5c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.755859 20563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21eaf12e87504154aa74e46187b415ae. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 13:57:54.757124 20541 leader_election.cc:343] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.747902 20489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319db728a12e43e5808e70ff91e5155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.763075 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:54.764505 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:54.765903 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:54.769579 20691 tablet_bootstrap.cc:654] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.738721 20697 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:54.771471 20641 tablet_service.cc:1468] Processing CreateTablet for tablet b9f875f43094448cbafb11704a281c08 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20250831 13:57:54.772802 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9f875f43094448cbafb11704a281c08. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 13:57:54.775624 20616 leader_election.cc:343] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:54.776316 20616 leader_election.cc:343] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:54.776883 20616 leader_election.cc:343] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.778028 20577 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:54.770408 20492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29595dd2c044618bd6cb4517fbc5c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.758468 20560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79dc07c2112a46fe82d7250aa95bd6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.780840 20697 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:54.790141 20562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319db728a12e43e5808e70ff91e5155f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.790102 20700 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:54.790632 20700 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.792492 20700 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:54.794283 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:54.794948 20490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21eaf12e87504154aa74e46187b415ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.795679 20467 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:57:54.796460 20467 leader_election.cc:343] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.798914 20488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3ed7784853456bbe218b149399b557. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.798833 20700 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:54.799540 20700 raft_consensus.cc:695] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:54.800474 20700 consensus_queue.cc:237] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.789783 20697 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:54.804978 20640 tablet_service.cc:1468] Processing CreateTablet for tablet f29595dd2c044618bd6cb4517fbc5c7b (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20250831 13:57:54.805672 20567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7956988b57844cfad1028024ac745e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.806506 20564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e93846478efb47a1b008fc73d5d25914. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.806938 20640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f29595dd2c044618bd6cb4517fbc5c7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.809624 20485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e93846478efb47a1b008fc73d5d25914. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.811779 20483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9f875f43094448cbafb11704a281c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.813470 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:54.814172 20502 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:54.815587 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:54.816684 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:54.817114 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:54.817993 20616 leader_election.cc:343] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.821018 20617 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:57:54.822120 20617 leader_election.cc:343] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.822515 20617 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:54.823545 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:54.824165 20500 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:54.822638 20692 tablet_bootstrap.cc:654] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:54.831831 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:54.836004 20543 leader_election.cc:343] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.818532 20697 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:54.906790 20697 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:54.907274 20697 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:54.907688 20697 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:54.908358 20639 tablet_service.cc:1468] Processing CreateTablet for tablet 50faaafc366a4da79cca52e16429da72 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250831 13:57:54.909837 20639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50faaafc366a4da79cca52e16429da72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.871235 20486 tablet_service.cc:1468] Processing CreateTablet for tablet 8de1861ba5b5442381501e5b214a51f9 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250831 13:57:54.911639 20486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de1861ba5b5442381501e5b214a51f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.875924 20635 tablet_service.cc:1468] Processing CreateTablet for tablet 21eaf12e87504154aa74e46187b415ae (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20250831 13:57:54.915409 20635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21eaf12e87504154aa74e46187b415ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.879293 20632 tablet_service.cc:1468] Processing CreateTablet for tablet e93846478efb47a1b008fc73d5d25914 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20250831 13:57:54.879933 20636 tablet_service.cc:1468] Processing CreateTablet for tablet 8de1861ba5b5442381501e5b214a51f9 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20250831 13:57:54.880398 20563 tablet_service.cc:1468] Processing CreateTablet for tablet 1dd4b973f11240dc9b8c480b1a77ccb0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250831 13:57:54.881721 20637 tablet_service.cc:1468] Processing CreateTablet for tablet 1dd4b973f11240dc9b8c480b1a77ccb0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250831 13:57:54.887495 20561 tablet_service.cc:1468] Processing CreateTablet for tablet 20bf19e3f9b04d0b9d20ce81aa982894 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250831 13:57:54.919425 20561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20bf19e3f9b04d0b9d20ce81aa982894. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.841913 20543 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:54.891376 20560 tablet_service.cc:1468] Processing CreateTablet for tablet a05a337dd9e04dec9e8874b1d6c4d114 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250831 13:57:54.890008 20564 tablet_service.cc:1468] Processing CreateTablet for tablet 50faaafc366a4da79cca52e16429da72 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250831 13:57:54.897913 20491 tablet_service.cc:1468] Processing CreateTablet for tablet 1dd4b973f11240dc9b8c480b1a77ccb0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20250831 13:57:54.900007 20492 tablet_service.cc:1468] Processing CreateTablet for tablet 20bf19e3f9b04d0b9d20ce81aa982894 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250831 13:57:54.900626 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:54.901237 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:54.942834 20634 tablet_service.cc:1468] Processing CreateTablet for tablet a05a337dd9e04dec9e8874b1d6c4d114 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250831 13:57:54.944800 20693 tablet_bootstrap.cc:492] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
W20250831 13:57:54.949669 20466 leader_election.cc:343] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.952383 20564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50faaafc366a4da79cca52e16429da72. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 13:57:54.931322 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 109239us
I20250831 13:57:54.950120 20491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dd4b973f11240dc9b8c480b1a77ccb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.950350 20632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e93846478efb47a1b008fc73d5d25914. 1 dirs total, 0 dirs full, 0 dirs failed
W20250831 13:57:54.939759 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 103966us
I20250831 13:57:54.955257 20398 catalog_manager.cc:5582] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:54.956252 20560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05a337dd9e04dec9e8874b1d6c4d114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.958638 20492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20bf19e3f9b04d0b9d20ce81aa982894. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.936894 20699 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:54.959682 20637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dd4b973f11240dc9b8c480b1a77ccb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.962785 20636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de1861ba5b5442381501e5b214a51f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.963198 20563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1dd4b973f11240dc9b8c480b1a77ccb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.965036 20487 tablet_service.cc:1468] Processing CreateTablet for tablet 50faaafc366a4da79cca52e16429da72 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20250831 13:57:54.966418 20487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50faaafc366a4da79cca52e16429da72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.971279 20638 tablet_service.cc:1468] Processing CreateTablet for tablet 20bf19e3f9b04d0b9d20ce81aa982894 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20250831 13:57:54.972595 20638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20bf19e3f9b04d0b9d20ce81aa982894. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.974810 20717 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:54.979748 20691 tablet_bootstrap.cc:492] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:54.981287 20693 ts_tablet_manager.cc:1397] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.362s user 0.020s sys 0.024s
W20250831 13:57:54.980191 20617 leader_election.cc:343] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:54.986168 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:54.986845 20567 tablet_service.cc:1468] Processing CreateTablet for tablet d1aba213b27e4f4aba685d97e9ba1626 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250831 13:57:54.987450 20617 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:54.988027 20567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1aba213b27e4f4aba685d97e9ba1626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.988348 20720 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:54.988974 20720 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:54.989261 20720 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:54.989527 20720 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:54.991650 20692 tablet_bootstrap.cc:492] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:54.992071 20692 ts_tablet_manager.cc:1397] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.410s user 0.016s sys 0.016s
I20250831 13:57:54.995769 20641 tablet_service.cc:1468] Processing CreateTablet for tablet d1aba213b27e4f4aba685d97e9ba1626 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250831 13:57:54.997040 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1aba213b27e4f4aba685d97e9ba1626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.000159 20697 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.001296 20650 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:55.003269 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:54.982585 20691 ts_tablet_manager.cc:1397] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.260s user 0.016s sys 0.020s
W20250831 13:57:55.004699 20616 leader_election.cc:343] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.005285 20697 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.005499 20466 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:55.006649 20700 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:54.981570 20488 tablet_service.cc:1468] Processing CreateTablet for tablet a05a337dd9e04dec9e8874b1d6c4d114 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20250831 13:57:55.010349 20692 raft_consensus.cc:357] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:55.011574 20692 raft_consensus.cc:383] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.011864 20692 raft_consensus.cc:738] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.012365 20565 tablet_service.cc:1468] Processing CreateTablet for tablet 5040bfaecba143f4af6334f314734a46 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250831 13:57:55.013340 20700 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:55.013617 20565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5040bfaecba143f4af6334f314734a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.016410 20692 consensus_queue.cc:260] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:55.019125 20720 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.020174 20562 tablet_service.cc:1468] Processing CreateTablet for tablet e9d8af1af01e4b7f83bc2aaded81ccbf (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250831 13:57:55.020952 20720 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.022034 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:57:55.023350 20616 leader_election.cc:343] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.024489 20562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d8af1af01e4b7f83bc2aaded81ccbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.025249 20491 tablet_service.cc:1468] Processing CreateTablet for tablet d1aba213b27e4f4aba685d97e9ba1626 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20250831 13:57:55.027122 20641 tablet_service.cc:1468] Processing CreateTablet for tablet e9d8af1af01e4b7f83bc2aaded81ccbf (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250831 13:57:55.028551 20691 raft_consensus.cc:357] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.029228 20691 raft_consensus.cc:383] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.029501 20691 raft_consensus.cc:738] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.029716 20566 tablet_service.cc:1468] Processing CreateTablet for tablet 5ce6782647ef46f095f726e552f5ad23 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250831 13:57:55.030910 20566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce6782647ef46f095f726e552f5ad23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:54.985287 20693 raft_consensus.cc:357] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.037109 20560 tablet_service.cc:1468] Processing CreateTablet for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250831 13:57:55.039466 20560 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.030164 20691 consensus_queue.cc:260] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.050993 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:55.051234 20488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05a337dd9e04dec9e8874b1d6c4d114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.051275 20640 tablet_service.cc:1468] Processing CreateTablet for tablet 5040bfaecba143f4af6334f314734a46 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250831 13:57:55.053767 20691 ts_tablet_manager.cc:1428] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20250831 13:57:55.054129 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:55.054598 20491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1aba213b27e4f4aba685d97e9ba1626. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.063721 20501 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.064213 20692 ts_tablet_manager.cc:1428] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.072s user 0.008s sys 0.000s
I20250831 13:57:55.065372 20692 tablet_bootstrap.cc:492] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.065471 20691 tablet_bootstrap.cc:492] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.055958 20564 tablet_service.cc:1468] Processing CreateTablet for tablet 346d047c73c648a5aab2f44efa73aee7 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250831 13:57:55.067011 20636 tablet_service.cc:1468] Processing CreateTablet for tablet 5ce6782647ef46f095f726e552f5ad23 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250831 13:57:55.062709 20558 tablet_service.cc:1468] Processing CreateTablet for tablet d9ed9ad66d8047f4af53abfa0192ff5a (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250831 13:57:55.067390 20564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346d047c73c648a5aab2f44efa73aee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.015463 20633 tablet_service.cc:1468] Processing CreateTablet for tablet 346d047c73c648a5aab2f44efa73aee7 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250831 13:57:55.082392 20558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ed9ad66d8047f4af53abfa0192ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.068774 20639 tablet_service.cc:1468] Processing CreateTablet for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250831 13:57:55.067292 20634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a05a337dd9e04dec9e8874b1d6c4d114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.085179 20639 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.069748 20483 tablet_service.cc:1468] Processing CreateTablet for tablet 5040bfaecba143f4af6334f314734a46 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20250831 13:57:55.034545 20693 raft_consensus.cc:383] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.071290 20635 tablet_service.cc:1468] Processing CreateTablet for tablet d9ed9ad66d8047f4af53abfa0192ff5a (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250831 13:57:55.077509 20484 tablet_service.cc:1468] Processing CreateTablet for tablet e9d8af1af01e4b7f83bc2aaded81ccbf (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20250831 13:57:55.089033 20693 raft_consensus.cc:738] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.089555 20640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5040bfaecba143f4af6334f314734a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.090934 20483 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5040bfaecba143f4af6334f314734a46. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.090818 20501 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:55.092438 20490 tablet_service.cc:1468] Processing CreateTablet for tablet 5ce6782647ef46f095f726e552f5ad23 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20250831 13:57:55.109381 20641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d8af1af01e4b7f83bc2aaded81ccbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.109942 20617 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:55.111167 20633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346d047c73c648a5aab2f44efa73aee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.111223 20720 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.111974 20720 raft_consensus.cc:695] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:55.114904 20636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce6782647ef46f095f726e552f5ad23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.115455 20720 consensus_queue.cc:237] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:55.139222 20448 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:57:55.143077 20693 consensus_queue.cc:260] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.145598 20693 ts_tablet_manager.cc:1428] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.163s user 0.004s sys 0.004s
I20250831 13:57:55.155164 20635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ed9ad66d8047f4af53abfa0192ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.159937 20692 tablet_bootstrap.cc:654] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.163312 20484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d8af1af01e4b7f83bc2aaded81ccbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.163967 20487 tablet_service.cc:1468] Processing CreateTablet for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20250831 13:57:55.165277 20487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.169286 20489 tablet_service.cc:1468] Processing CreateTablet for tablet 346d047c73c648a5aab2f44efa73aee7 (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20250831 13:57:55.171561 20693 tablet_bootstrap.cc:492] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:55.175105 20489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346d047c73c648a5aab2f44efa73aee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.191282 20500 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:55.192656 20617 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:55.193696 20691 tablet_bootstrap.cc:654] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.194020 20720 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.194744 20720 raft_consensus.cc:695] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:55.195607 20720 consensus_queue.cc:237] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.217002 20699 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.217510 20699 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:55.219028 20692 tablet_bootstrap.cc:492] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.219444 20699 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:57:55.220757 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:55.220875 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.223425 20650 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:55.225334 20541 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:57:55.226262 20543 leader_election.cc:343] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.227497 20699 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:55.227867 20699 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:55.228273 20699 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
W20250831 13:57:55.232451 20466 consensus_peers.cc:489] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 -> Peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Couldn't send request to peer 44d163094add49049cc749547d891602. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:57:55.237735 20490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce6782647ef46f095f726e552f5ad23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.238706 20692 ts_tablet_manager.cc:1397] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.174s user 0.009s sys 0.007s
I20250831 13:57:55.247807 20399 catalog_manager.cc:5582] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:55.264331 20693 tablet_bootstrap.cc:654] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.273278 20486 tablet_service.cc:1468] Processing CreateTablet for tablet d9ed9ad66d8047f4af53abfa0192ff5a (DEFAULT_TABLE table=test-workload [id=bf9639b2cb72453d8b67d48646dc2938]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20250831 13:57:55.273996 20691 tablet_bootstrap.cc:492] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.274619 20691 ts_tablet_manager.cc:1397] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.209s user 0.022s sys 0.030s
I20250831 13:57:55.280861 20692 raft_consensus.cc:357] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.287079 20692 raft_consensus.cc:383] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.287495 20692 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.288383 20692 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.278415 20691 raft_consensus.cc:357] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.278245 20699 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:55.290019 20691 raft_consensus.cc:383] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.282852 20486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ed9ad66d8047f4af53abfa0192ff5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:57:55.290490 20691 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.299742 20691 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.302327 20691 ts_tablet_manager.cc:1428] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250831 13:57:55.302639 20700 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:55.305217 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:55.305850 20650 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.309962 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:57:55.311414 20543 leader_election.cc:343] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.313047 20691 tablet_bootstrap.cc:492] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.312165 20699 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:57:55.319406 20692 ts_tablet_manager.cc:1428] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.077s user 0.010s sys 0.006s
I20250831 13:57:55.320494 20692 tablet_bootstrap.cc:492] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.323786 20691 tablet_bootstrap.cc:654] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.323750 20398 catalog_manager.cc:5582] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:55.328991 20692 tablet_bootstrap.cc:654] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.330387 20693 tablet_bootstrap.cc:492] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:55.330909 20693 ts_tablet_manager.cc:1397] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.160s user 0.023s sys 0.019s
I20250831 13:57:55.334535 20693 raft_consensus.cc:357] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.335569 20693 raft_consensus.cc:383] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.335953 20693 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.337150 20693 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.344286 20693 ts_tablet_manager.cc:1428] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20250831 13:57:55.344877 20650 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:55.346333 20541 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.347604 20691 tablet_bootstrap.cc:492] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.347605 20734 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.348347 20691 ts_tablet_manager.cc:1397] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.036s user 0.016s sys 0.014s
I20250831 13:57:55.348452 20734 raft_consensus.cc:695] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:55.349349 20734 consensus_queue.cc:237] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:55.351362 20691 raft_consensus.cc:357] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.352075 20691 raft_consensus.cc:383] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.352416 20691 raft_consensus.cc:738] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.353241 20691 consensus_queue.cc:260] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.354208 20692 tablet_bootstrap.cc:492] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.354691 20692 ts_tablet_manager.cc:1397] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.008s
I20250831 13:57:55.355682 20691 ts_tablet_manager.cc:1428] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250831 13:57:55.356709 20691 tablet_bootstrap.cc:492] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.357344 20692 raft_consensus.cc:357] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.358045 20692 raft_consensus.cc:383] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.358350 20692 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.359520 20692 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.362099 20692 ts_tablet_manager.cc:1428] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:57:55.363085 20692 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.363543 20691 tablet_bootstrap.cc:654] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.375213 20692 tablet_bootstrap.cc:654] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.375820 20693 tablet_bootstrap.cc:492] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:55.385090 20691 tablet_bootstrap.cc:492] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.385632 20691 ts_tablet_manager.cc:1397] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.016s
I20250831 13:57:55.388661 20691 raft_consensus.cc:357] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.389374 20691 raft_consensus.cc:383] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.389678 20691 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.390313 20691 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.387421 20693 tablet_bootstrap.cc:654] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.395280 20691 ts_tablet_manager.cc:1428] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250831 13:57:55.396346 20691 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.399067 20720 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.399680 20692 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.399524 20720 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.400161 20692 ts_tablet_manager.cc:1397] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.005s
I20250831 13:57:55.401589 20720 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.403152 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.403784 20500 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:55.404721 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:55.405941 20616 leader_election.cc:343] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.406541 20691 tablet_bootstrap.cc:654] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.406827 20617 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:55.407908 20720 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:55.408249 20720 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:55.408557 20720 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.409961 20398 catalog_manager.cc:5582] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:55.411484 20691 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.411779 20691 ts_tablet_manager.cc:1397] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20250831 13:57:55.413769 20691 raft_consensus.cc:357] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.414359 20691 raft_consensus.cc:383] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.414598 20691 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.414798 20692 raft_consensus.cc:357] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.415351 20691 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.415729 20692 raft_consensus.cc:383] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.416214 20692 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.417557 20691 ts_tablet_manager.cc:1428] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:57:55.418510 20691 tablet_bootstrap.cc:492] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.417021 20692 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.421437 20692 ts_tablet_manager.cc:1428] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.021s user 0.003s sys 0.004s
I20250831 13:57:55.422633 20692 tablet_bootstrap.cc:492] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.423676 20720 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.424237 20691 tablet_bootstrap.cc:654] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.427217 20693 tablet_bootstrap.cc:492] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:55.435266 20692 tablet_bootstrap.cc:654] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.437805 20693 ts_tablet_manager.cc:1397] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.012s
I20250831 13:57:55.439013 20720 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.439438 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:55.439924 20500 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.440567 20693 raft_consensus.cc:357] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.439870 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:55.441298 20693 raft_consensus.cc:383] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.441608 20693 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
W20250831 13:57:55.442261 20616 leader_election.cc:343] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.442382 20693 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.445025 20500 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:55.445271 20693 ts_tablet_manager.cc:1428] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:57:55.446108 20617 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:55.446597 20693 tablet_bootstrap.cc:492] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:55.447295 20720 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.447989 20720 raft_consensus.cc:695] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:55.448897 20720 consensus_queue.cc:237] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.455040 20693 tablet_bootstrap.cc:654] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.461723 20692 tablet_bootstrap.cc:492] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.462232 20692 ts_tablet_manager.cc:1397] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.013s
I20250831 13:57:55.464742 20692 raft_consensus.cc:357] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.465400 20692 raft_consensus.cc:383] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.465741 20692 raft_consensus.cc:738] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.466427 20692 consensus_queue.cc:260] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.467494 20691 tablet_bootstrap.cc:492] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.468011 20691 ts_tablet_manager.cc:1397] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.007s
I20250831 13:57:55.470450 20691 raft_consensus.cc:357] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.461723 20693 tablet_bootstrap.cc:492] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:55.471115 20691 raft_consensus.cc:383] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.471506 20693 ts_tablet_manager.cc:1397] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.004s
I20250831 13:57:55.471460 20691 raft_consensus.cc:738] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.472229 20691 consensus_queue.cc:260] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.474324 20691 ts_tablet_manager.cc:1428] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250831 13:57:55.474169 20693 raft_consensus.cc:357] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.474890 20693 raft_consensus.cc:383] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.475217 20693 raft_consensus.cc:738] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.475319 20691 tablet_bootstrap.cc:492] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.475898 20693 consensus_queue.cc:260] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.470798 20692 ts_tablet_manager.cc:1428] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.008s user 0.000s sys 0.007s
I20250831 13:57:55.477563 20692 tablet_bootstrap.cc:492] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.480592 20691 tablet_bootstrap.cc:654] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.480496 20400 catalog_manager.cc:5582] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:55.483631 20692 tablet_bootstrap.cc:654] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.487694 20693 ts_tablet_manager.cc:1428] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20250831 13:57:55.495038 20717 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.495678 20717 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.498335 20717 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.500147 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:55.501036 20574 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:55.502545 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.504853 20616 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:57:55.506664 20617 leader_election.cc:343] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.508042 20717 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:55.508615 20717 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:55.509181 20717 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.509398 20693 tablet_bootstrap.cc:492] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:55.517185 20692 tablet_bootstrap.cc:492] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.517668 20692 ts_tablet_manager.cc:1397] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.022s
I20250831 13:57:55.520059 20692 raft_consensus.cc:357] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.520721 20692 raft_consensus.cc:383] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.520959 20692 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.521672 20692 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.518656 20693 tablet_bootstrap.cc:654] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.524009 20692 ts_tablet_manager.cc:1428] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250831 13:57:55.524950 20692 tablet_bootstrap.cc:492] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.532717 20692 tablet_bootstrap.cc:654] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.534931 20693 tablet_bootstrap.cc:492] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:55.535557 20693 ts_tablet_manager.cc:1397] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.010s
I20250831 13:57:55.539178 20693 raft_consensus.cc:357] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:55.540172 20693 raft_consensus.cc:383] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.540573 20693 raft_consensus.cc:738] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.541518 20693 consensus_queue.cc:260] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:57:55.543637 20693 ts_tablet_manager.cc:1428] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250831 13:57:55.544898 20717 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.545316 20693 tablet_bootstrap.cc:492] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:55.545593 20692 tablet_bootstrap.cc:492] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.546026 20692 ts_tablet_manager.cc:1397] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.000s
I20250831 13:57:55.546823 20717 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.548326 20692 raft_consensus.cc:357] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.548837 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:55.549095 20692 raft_consensus.cc:383] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.549422 20692 raft_consensus.cc:738] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.549986 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:55.550547 20574 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.550096 20692 consensus_queue.cc:260] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.551147 20691 tablet_bootstrap.cc:492] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.551592 20691 ts_tablet_manager.cc:1397] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.076s user 0.037s sys 0.031s
I20250831 13:57:55.552577 20692 ts_tablet_manager.cc:1428] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250831 13:57:55.553787 20692 tablet_bootstrap.cc:492] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.553951 20691 raft_consensus.cc:357] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.554605 20691 raft_consensus.cc:383] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.554903 20691 raft_consensus.cc:738] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
W20250831 13:57:55.555245 20617 leader_election.cc:343] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.555516 20691 consensus_queue.cc:260] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.556037 20693 tablet_bootstrap.cc:654] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.557883 20691 ts_tablet_manager.cc:1428] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250831 13:57:55.559720 20691 tablet_bootstrap.cc:492] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.560158 20734 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.560614 20734 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.561375 20574 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:55.562880 20692 tablet_bootstrap.cc:654] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.562597 20616 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.563789 20717 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.563856 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.564409 20717 raft_consensus.cc:695] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:55.565076 20691 tablet_bootstrap.cc:654] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
W20250831 13:57:55.565295 20543 leader_election.cc:343] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.565287 20717 consensus_queue.cc:237] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.565306 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:55.566607 20650 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:55.567854 20541 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.562707 20734 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:55.569365 20699 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:55.569829 20699 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:55.570245 20699 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.574170 20717 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.574699 20717 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.577999 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.578575 20717 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.593664 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:55.595244 20616 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.596720 20717 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:55.597450 20699 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.601035 20692 tablet_bootstrap.cc:492] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.601234 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:55.601483 20692 ts_tablet_manager.cc:1397] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.005s
W20250831 13:57:55.602591 20543 leader_election.cc:343] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.604125 20692 raft_consensus.cc:357] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.604744 20692 raft_consensus.cc:383] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.605006 20692 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.606163 20693 tablet_bootstrap.cc:492] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:55.606956 20693 ts_tablet_manager.cc:1397] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.062s user 0.018s sys 0.015s
I20250831 13:57:55.606871 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:55.607475 20650 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.608332 20691 tablet_bootstrap.cc:492] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.608747 20691 ts_tablet_manager.cc:1397] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.049s user 0.018s sys 0.028s
I20250831 13:57:55.608911 20693 raft_consensus.cc:357] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.609416 20693 raft_consensus.cc:383] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.609604 20693 raft_consensus.cc:738] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.610240 20693 consensus_queue.cc:260] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.611986 20693 ts_tablet_manager.cc:1428] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250831 13:57:55.612900 20693 tablet_bootstrap.cc:492] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
W20250831 13:57:55.614197 20616 consensus_peers.cc:489] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 -> Peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Couldn't send request to peer 212b66976f874aa682cae0fb77928786. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:57:55.611168 20691 raft_consensus.cc:357] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.614917 20691 raft_consensus.cc:383] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.615206 20691 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.615784 20691 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.618144 20691 ts_tablet_manager.cc:1428] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250831 13:57:55.618552 20693 tablet_bootstrap.cc:654] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.619107 20691 tablet_bootstrap.cc:492] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.622942 20650 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:55.624163 20541 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.623584 20399 catalog_manager.cc:5582] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:55.625054 20693 tablet_bootstrap.cc:492] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:55.625285 20691 tablet_bootstrap.cc:654] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.605638 20692 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.625098 20734 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.625828 20693 ts_tablet_manager.cc:1397] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.001s
I20250831 13:57:55.627492 20692 ts_tablet_manager.cc:1428] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20250831 13:57:55.628717 20692 tablet_bootstrap.cc:492] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.631999 20693 raft_consensus.cc:357] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.632712 20693 raft_consensus.cc:383] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.633024 20693 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.633919 20693 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.635367 20692 tablet_bootstrap.cc:654] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.639770 20693 ts_tablet_manager.cc:1428] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250831 13:57:55.640743 20693 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:55.646548 20693 tablet_bootstrap.cc:654] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.648378 20717 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.648859 20717 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.652698 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.653529 20500 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:55.654553 20720 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.654598 20734 raft_consensus.cc:695] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:55.654883 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:55.655143 20720 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.655567 20734 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.656641 20617 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:55.657004 20720 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.657565 20720 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:55.657872 20717 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.659420 20693 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
W20250831 13:57:55.659662 20616 leader_election.cc:343] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.659922 20693 ts_tablet_manager.cc:1397] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.002s
I20250831 13:57:55.660705 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:55.661329 20574 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:55.662596 20616 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:55.662837 20693 raft_consensus.cc:357] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.663620 20693 raft_consensus.cc:383] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.663657 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.664058 20693 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.664343 20500 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:55.664966 20693 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.657891 20720 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:55.666240 20720 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.670897 20717 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:55.671288 20717 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:55.671492 20693 ts_tablet_manager.cc:1428] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250831 13:57:55.672652 20717 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.674520 20700 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.675031 20700 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.676976 20700 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:55.605706 20699 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:55.678962 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:55.679572 20650 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:55.681437 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:55.681713 20466 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:55.682071 20574 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:55.681160 20397 catalog_manager.cc:5582] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:55.687472 20700 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:55.687942 20700 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:55.688377 20700 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.692515 20720 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.694761 20693 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:55.695443 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:55.696022 20500 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.697448 20700 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.700340 20700 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:55.701100 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:55.701438 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:55.701706 20574 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.701997 20650 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.705577 20577 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.709079 20650 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:55.710207 20466 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:55.711126 20700 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.711620 20700 raft_consensus.cc:695] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:55.712466 20700 consensus_queue.cc:237] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.718884 20697 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:55.722334 20693 tablet_bootstrap.cc:654] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.723740 20500 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:55.732815 20691 tablet_bootstrap.cc:492] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.733419 20691 ts_tablet_manager.cc:1397] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.115s user 0.030s sys 0.037s
I20250831 13:57:55.734637 20617 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:55.736994 20692 tablet_bootstrap.cc:492] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.736979 20767 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.737579 20692 ts_tablet_manager.cc:1397] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.109s user 0.028s sys 0.039s
I20250831 13:57:55.739015 20767 raft_consensus.cc:695] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:55.739588 20717 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.740345 20697 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:55.740712 20692 raft_consensus.cc:357] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.741524 20692 raft_consensus.cc:383] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.740198 20767 consensus_queue.cc:237] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.742660 20574 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:55.745569 20742 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.745805 20691 raft_consensus.cc:357] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.746049 20742 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.741873 20692 raft_consensus.cc:738] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.749092 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:55.749315 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:55.749719 20577 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:55.749902 20650 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:55.749497 20692 consensus_queue.cc:260] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.750821 20467 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:55.751071 20742 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:55.746461 20691 raft_consensus.cc:383] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.751649 20691 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.752251 20742 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:55.752629 20742 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:55.752405 20691 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.752964 20742 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.754035 20692 ts_tablet_manager.cc:1428] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20250831 13:57:55.755101 20692 tablet_bootstrap.cc:492] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.756597 20691 ts_tablet_manager.cc:1428] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20250831 13:57:55.757633 20691 tablet_bootstrap.cc:492] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:55.761006 20742 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.764073 20697 consensus_queue.cc:1035] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:55.764793 20734 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.765420 20734 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.767875 20734 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:55.769927 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:57:55.771799 20541 leader_election.cc:343] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.769917 20691 tablet_bootstrap.cc:654] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.772851 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:57:55.774247 20616 leader_election.cc:343] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.775023 20734 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.775655 20734 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.780503 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:55.782186 20734 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:57:55.784368 20541 leader_election.cc:343] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.789856 20692 tablet_bootstrap.cc:654] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.794023 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.794595 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.796712 20499 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.799783 20692 tablet_bootstrap.cc:492] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.800347 20692 ts_tablet_manager.cc:1397] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.045s user 0.009s sys 0.003s
W20250831 13:57:55.803236 20543 leader_election.cc:343] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.803057 20692 raft_consensus.cc:357] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.803897 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.804281 20692 raft_consensus.cc:383] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.804605 20692 raft_consensus.cc:738] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.805301 20692 consensus_queue.cc:260] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:55.806313 20543 leader_election.cc:343] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.806893 20543 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.807753 20692 ts_tablet_manager.cc:1428] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 13:57:55.808027 20759 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:55.808851 20692 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.808917 20759 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:55.810153 20759 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.810590 20759 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.812968 20759 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:55.814966 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:55.817826 20692 tablet_bootstrap.cc:654] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.824424 20734 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:55.832868 20717 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.834017 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:55.834848 20577 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.839434 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:55.840183 20500 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.841163 20720 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.843601 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.860389 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:55.861059 20575 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.865689 20499 raft_consensus.cc:2391] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:55.872061 20693 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:55.873070 20500 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:55.873735 20543 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.875984 20693 ts_tablet_manager.cc:1397] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.204s user 0.021s sys 0.029s
I20250831 13:57:55.878211 20691 tablet_bootstrap.cc:492] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:55.878652 20691 ts_tablet_manager.cc:1397] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.121s user 0.019s sys 0.029s
I20250831 13:57:55.880380 20734 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.882810 20575 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:55.877794 20767 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.887176 20617 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:55.884086 20742 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:57:55.890815 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 139887us
I20250831 13:57:55.891076 20699 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:55.891600 20775 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.892046 20775 raft_consensus.cc:695] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:55.892841 20775 consensus_queue.cc:237] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.900002 20693 raft_consensus.cc:357] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.887189 20767 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.900918 20693 raft_consensus.cc:383] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.901203 20697 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.901307 20693 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.901692 20697 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.902200 20693 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.903374 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.903602 20697 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.906642 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:55.906642 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:55.906975 20767 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.907302 20651 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.891299 20734 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 13:57:55.907744 20617 leader_election.cc:343] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.908818 20693 ts_tablet_manager.cc:1428] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20250831 13:57:55.909924 20693 tablet_bootstrap.cc:492] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:55.909742 20467 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:55.910252 20577 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
W20250831 13:57:55.913357 20617 leader_election.cc:343] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.917451 20691 raft_consensus.cc:357] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.917872 20691 raft_consensus.cc:383] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.918064 20691 raft_consensus.cc:738] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.914073 20700 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:55.919382 20691 consensus_queue.cc:260] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.919123 20700 raft_consensus.cc:695] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:55.928754 20692 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:55.929064 20693 tablet_bootstrap.cc:654] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:55.929445 20692 ts_tablet_manager.cc:1397] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.121s user 0.009s sys 0.055s
I20250831 13:57:55.919135 20649 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:55.934000 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:55.936808 20692 raft_consensus.cc:357] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.937259 20692 raft_consensus.cc:383] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:55.937554 20692 raft_consensus.cc:738] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:55.938053 20692 consensus_queue.cc:260] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.939640 20692 ts_tablet_manager.cc:1428] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.010s user 0.003s sys 0.001s
I20250831 13:57:55.939081 20397 catalog_manager.cc:5582] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:55.940434 20692 tablet_bootstrap.cc:492] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:55.942950 20651 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:55.944051 20700 consensus_queue.cc:237] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.942874 20720 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:55.946197 20691 ts_tablet_manager.cc:1428] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.067s user 0.003s sys 0.003s
I20250831 13:57:55.929926 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:55.942525 20616 leader_election.cc:343] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.947149 20616 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:55.948675 20720 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:55.967206 20720 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.973857 20720 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:55.976322 20720 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:55.977010 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:55.983404 20617 leader_election.cc:343] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:55.996518 20742 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:55.999351 20742 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.001241 20742 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.001782 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.002708 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:56.003296 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 57320.4us
W20250831 13:57:56.006968 20616 leader_election.cc:343] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.007282 20691 tablet_bootstrap.cc:492] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.007411 20616 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.007211 20576 raft_consensus.cc:2391] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:56.003607 20692 tablet_bootstrap.cc:654] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.009474 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.011822 20466 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:56.012841 20742 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.015303 20720 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:56.022271 20742 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 13:57:56.023473 20616 leader_election.cc:343] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.023918 20717 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:56.024333 20691 tablet_bootstrap.cc:654] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.023919 20616 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.028249 20717 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:56.024896 20398 catalog_manager.cc:5582] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:56.043552 20691 tablet_bootstrap.cc:492] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.044164 20691 ts_tablet_manager.cc:1397] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.001s
I20250831 13:57:56.044612 20577 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.047300 20691 raft_consensus.cc:357] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.048184 20691 raft_consensus.cc:383] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.048511 20691 raft_consensus.cc:738] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.049638 20691 consensus_queue.cc:260] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.052721 20691 ts_tablet_manager.cc:1428] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20250831 13:57:56.053885 20691 tablet_bootstrap.cc:492] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.060539 20693 tablet_bootstrap.cc:492] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.061203 20693 ts_tablet_manager.cc:1397] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.151s user 0.019s sys 0.014s
I20250831 13:57:56.068048 20691 tablet_bootstrap.cc:654] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.068346 20693 raft_consensus.cc:357] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.069114 20693 raft_consensus.cc:383] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.069442 20693 raft_consensus.cc:738] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.070093 20693 consensus_queue.cc:260] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.071466 20692 tablet_bootstrap.cc:492] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.071914 20692 ts_tablet_manager.cc:1397] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.132s user 0.008s sys 0.062s
I20250831 13:57:56.072722 20693 ts_tablet_manager.cc:1428] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20250831 13:57:56.073621 20693 tablet_bootstrap.cc:492] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.073459 20397 catalog_manager.cc:5582] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:56.074720 20692 raft_consensus.cc:357] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.075364 20692 raft_consensus.cc:383] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.075635 20692 raft_consensus.cc:738] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.076288 20692 consensus_queue.cc:260] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.078586 20692 ts_tablet_manager.cc:1428] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250831 13:57:56.079437 20692 tablet_bootstrap.cc:492] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.079914 20693 tablet_bootstrap.cc:654] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.086287 20692 tablet_bootstrap.cc:654] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.092207 20775 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.092674 20775 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.093292 20691 tablet_bootstrap.cc:492] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.093678 20691 ts_tablet_manager.cc:1397] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.040s user 0.022s sys 0.007s
I20250831 13:57:56.095340 20502 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.095844 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:56.096532 20691 raft_consensus.cc:357] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:56.097051 20617 leader_election.cc:343] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.097141 20691 raft_consensus.cc:383] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.097546 20691 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.094556 20775 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:56.098220 20691 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.098644 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:56.100281 20616 leader_election.cc:343] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.100811 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.101766 20693 tablet_bootstrap.cc:492] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.102212 20775 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:56.102356 20693 ts_tablet_manager.cc:1397] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.008s
I20250831 13:57:56.104658 20691 ts_tablet_manager.cc:1428] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20250831 13:57:56.105464 20691 tablet_bootstrap.cc:492] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.110189 20700 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.110617 20700 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.111465 20734 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:56.113906 20693 raft_consensus.cc:357] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.115401 20693 raft_consensus.cc:383] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.115711 20693 raft_consensus.cc:738] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.115851 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.116420 20649 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.116626 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.117534 20466 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:56.118125 20700 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.118150 20742 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:57:56.118726 20742 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.119055 20742 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:56.123118 20734 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:56.116430 20693 consensus_queue.cc:260] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.132681 20742 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.133915 20502 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250831 13:57:56.136586 20734 consensus_queue.cc:1046] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:56.136965 20693 ts_tablet_manager.cc:1428] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20250831 13:57:56.137442 20734 raft_consensus.cc:3053] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:57:56.137800 20734 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:56.138167 20693 tablet_bootstrap.cc:492] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.138352 20775 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:56.138542 20734 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.139601 20734 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:57:56.141844 20699 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.144366 20742 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.144977 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:56.142452 20699 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.145356 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:56.146739 20692 tablet_bootstrap.cc:492] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.149557 20692 ts_tablet_manager.cc:1397] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.070s user 0.032s sys 0.027s
I20250831 13:57:56.151135 20691 tablet_bootstrap.cc:654] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.151216 20699 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.154475 20699 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.155050 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:56.153594 20398 catalog_manager.cc:5582] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
W20250831 13:57:56.156610 20448 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:57:56.156692 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:57:56.157660 20543 leader_election.cc:343] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.158190 20775 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:56.160334 20577 raft_consensus.cc:2408] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250831 13:57:56.161092 20692 raft_consensus.cc:357] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.154920 20699 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.161720 20692 raft_consensus.cc:383] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.162206 20692 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.163683 20692 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.162906 20467 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:57:56.166509 20541 leader_election.cc:343] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.166908 20699 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.166954 20541 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.168385 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:56.169574 20543 leader_election.cc:343] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.169910 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.167423 20742 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:57:56.171634 20692 ts_tablet_manager.cc:1428] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.022s user 0.004s sys 0.002s
I20250831 13:57:56.172387 20699 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 13:57:56.172662 20541 leader_election.cc:343] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.173096 20541 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.174242 20699 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:56.176359 20693 tablet_bootstrap.cc:654] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.185150 20692 tablet_bootstrap.cc:492] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.191461 20692 tablet_bootstrap.cc:654] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.192171 20693 tablet_bootstrap.cc:492] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.192648 20693 ts_tablet_manager.cc:1397] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.002s
I20250831 13:57:56.195123 20693 raft_consensus.cc:357] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.195577 20693 raft_consensus.cc:383] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.195789 20693 raft_consensus.cc:738] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.196262 20693 consensus_queue.cc:260] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.197705 20693 ts_tablet_manager.cc:1428] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250831 13:57:56.198343 20693 tablet_bootstrap.cc:492] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.203078 20693 tablet_bootstrap.cc:654] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.221388 20742 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.221880 20742 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.223863 20742 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.225842 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.225947 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:56.227844 20467 leader_election.cc:343] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:56.228633 20466 leader_election.cc:343] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.229177 20466 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:56.229823 20691 tablet_bootstrap.cc:492] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.230396 20691 ts_tablet_manager.cc:1397] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.125s user 0.015s sys 0.030s
I20250831 13:57:56.230515 20742 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:56.234277 20691 raft_consensus.cc:357] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.235464 20691 raft_consensus.cc:383] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.235857 20691 raft_consensus.cc:738] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.236850 20691 consensus_queue.cc:260] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.240046 20693 tablet_bootstrap.cc:492] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.240518 20693 ts_tablet_manager.cc:1397] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.024s
I20250831 13:57:56.241133 20692 tablet_bootstrap.cc:492] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.241499 20691 ts_tablet_manager.cc:1428] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250831 13:57:56.241539 20692 ts_tablet_manager.cc:1397] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.057s user 0.022s sys 0.022s
I20250831 13:57:56.243379 20691 tablet_bootstrap.cc:492] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.243978 20693 raft_consensus.cc:357] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.244652 20693 raft_consensus.cc:383] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.244499 20692 raft_consensus.cc:357] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.245003 20693 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.245172 20692 raft_consensus.cc:383] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.245579 20692 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.245994 20693 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.246274 20692 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.248283 20693 ts_tablet_manager.cc:1428] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250831 13:57:56.249133 20693 tablet_bootstrap.cc:492] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.249413 20691 tablet_bootstrap.cc:654] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.251916 20692 ts_tablet_manager.cc:1428] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.010s user 0.001s sys 0.004s
I20250831 13:57:56.252780 20692 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.255600 20693 tablet_bootstrap.cc:654] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.258031 20692 tablet_bootstrap.cc:654] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.260599 20691 tablet_bootstrap.cc:492] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.261121 20691 ts_tablet_manager.cc:1397] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20250831 13:57:56.264175 20691 raft_consensus.cc:357] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.264961 20691 raft_consensus.cc:383] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.265303 20691 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.266251 20691 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.269345 20692 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.269758 20692 ts_tablet_manager.cc:1397] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.002s
I20250831 13:57:56.272092 20692 raft_consensus.cc:357] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.272696 20692 raft_consensus.cc:383] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.272945 20692 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.273546 20692 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.274240 20691 ts_tablet_manager.cc:1428] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.013s user 0.008s sys 0.001s
I20250831 13:57:56.275493 20691 tablet_bootstrap.cc:492] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.275890 20692 ts_tablet_manager.cc:1428] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250831 13:57:56.276721 20692 tablet_bootstrap.cc:492] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.282158 20692 tablet_bootstrap.cc:654] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.283396 20691 tablet_bootstrap.cc:654] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.285233 20693 tablet_bootstrap.cc:492] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.285637 20693 ts_tablet_manager.cc:1397] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.010s
I20250831 13:57:56.288120 20693 raft_consensus.cc:357] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.288589 20692 tablet_bootstrap.cc:492] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.288754 20693 raft_consensus.cc:383] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.288980 20692 ts_tablet_manager.cc:1397] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20250831 13:57:56.289047 20693 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.289671 20693 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.290186 20742 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:56.291250 20692 raft_consensus.cc:357] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.292217 20692 raft_consensus.cc:383] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.292431 20692 raft_consensus.cc:738] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.292974 20692 consensus_queue.cc:260] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.295663 20692 ts_tablet_manager.cc:1428] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250831 13:57:56.296504 20692 tablet_bootstrap.cc:492] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.301328 20770 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.302178 20692 tablet_bootstrap.cc:654] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.301764 20770 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.308389 20574 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.309317 20770 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.309973 20700 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:56.311224 20700 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.311584 20700 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.313338 20700 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.313769 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.314518 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:57:56.316286 20466 leader_election.cc:343] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:56.316720 20467 leader_election.cc:343] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.316874 20767 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:56.317330 20693 ts_tablet_manager.cc:1428] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.031s user 0.006s sys 0.001s
I20250831 13:57:56.317268 20467 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:56.318297 20693 tablet_bootstrap.cc:492] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.318975 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.319541 20576 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:56.319708 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.320281 20649 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:56.324064 20770 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:56.325217 20775 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.325850 20775 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.326920 20693 tablet_bootstrap.cc:654] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.328322 20775 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:56.331606 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.331725 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:56.332393 20574 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
W20250831 13:57:56.333571 20617 leader_election.cc:343] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.334450 20616 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.336617 20767 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.337054 20767 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.337630 20767 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.341535 20693 tablet_bootstrap.cc:492] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.342063 20693 ts_tablet_manager.cc:1397] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.007s
I20250831 13:57:56.344713 20693 raft_consensus.cc:357] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.345445 20693 raft_consensus.cc:383] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.345769 20693 raft_consensus.cc:738] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.346459 20693 consensus_queue.cc:260] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.348784 20693 ts_tablet_manager.cc:1428] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250831 13:57:56.349071 20717 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:56.325198 20467 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:56.355942 20700 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.356422 20700 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.356626 20691 tablet_bootstrap.cc:492] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.357277 20700 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.357611 20691 ts_tablet_manager.cc:1397] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.082s user 0.018s sys 0.010s
I20250831 13:57:56.360037 20691 raft_consensus.cc:357] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.362812 20691 raft_consensus.cc:383] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.363515 20691 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.364614 20693 tablet_bootstrap.cc:492] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.362876 20742 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.365537 20742 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.369019 20742 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.369968 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.370779 20651 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:56.372110 20466 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:56.374240 20691 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.375965 20742 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.376413 20742 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.376808 20742 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.378161 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.380949 20691 ts_tablet_manager.cc:1428] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250831 13:57:56.370796 20693 tablet_bootstrap.cc:654] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
W20250831 13:57:56.380824 20467 leader_election.cc:343] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.383693 20691 tablet_bootstrap.cc:492] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.387684 20700 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.390777 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:56.391546 20574 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.394320 20691 tablet_bootstrap.cc:654] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.400913 20775 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:56.406293 20693 tablet_bootstrap.cc:492] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.406826 20693 ts_tablet_manager.cc:1397] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.004s
I20250831 13:57:56.400914 20767 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.422200 20767 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:56.442951 20574 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.443979 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:56.444713 20577 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.446072 20467 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:56.446615 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:56.434984 20700 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.440157 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:56.448189 20651 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
W20250831 13:57:56.455760 20617 leader_election.cc:343] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.463912 20742 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.465080 20734 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.465550 20734 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.466528 20742 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.467257 20742 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.467659 20742 raft_consensus.cc:695] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:56.468120 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:56.468606 20734 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.468627 20649 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.468400 20742 consensus_queue.cc:237] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.478394 20693 raft_consensus.cc:357] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.479300 20693 raft_consensus.cc:383] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.479629 20693 raft_consensus.cc:738] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.482892 20693 consensus_queue.cc:260] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.499025 20697 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.499631 20697 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.501941 20697 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:56.502763 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.508606 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:56.511391 20617 leader_election.cc:343] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:56.511603 20541 leader_election.cc:343] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.512414 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.480466 20767 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
W20250831 13:57:56.513599 20616 leader_election.cc:343] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.514003 20616 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.515301 20767 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:56.523392 20649 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.528373 20651 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.533821 20812 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.537045 20812 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.545641 20812 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.538187 20697 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:57:56.534144 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 88149.2us
I20250831 13:57:56.555056 20697 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.563859 20697 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:56.564921 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.565543 20574 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:56.566622 20616 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:56.567893 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:56.568598 20814 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.569237 20814 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.569967 20767 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.570461 20767 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.570914 20767 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.567663 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.572316 20573 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:56.567061 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:57:56.576388 20617 leader_election.cc:343] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.577044 20814 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.587397 20691 tablet_bootstrap.cc:492] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.587817 20691 ts_tablet_manager.cc:1397] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.204s user 0.069s sys 0.069s
I20250831 13:57:56.593600 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:56.594599 20502 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:56.595453 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.597626 20692 tablet_bootstrap.cc:492] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.598234 20692 ts_tablet_manager.cc:1397] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.302s user 0.066s sys 0.102s
I20250831 13:57:56.597705 20691 raft_consensus.cc:357] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.599365 20691 raft_consensus.cc:383] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.599537 20691 raft_consensus.cc:738] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.599992 20691 consensus_queue.cc:260] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.604239 20692 raft_consensus.cc:357] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.604861 20692 raft_consensus.cc:383] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.605145 20692 raft_consensus.cc:738] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.605970 20692 consensus_queue.cc:260] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.605686 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.607546 20649 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:56.607663 20467 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:57:56.610877 20467 leader_election.cc:343] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.610962 20818 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.611418 20818 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.612541 20818 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.612620 20817 consensus_queue.cc:1035] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:56.613209 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.613564 20767 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.615110 20649 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:56.619328 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:56.620010 20574 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.621836 20767 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:56.623255 20691 ts_tablet_manager.cc:1428] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.035s user 0.005s sys 0.001s
I20250831 13:57:56.623914 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:56.624045 20692 ts_tablet_manager.cc:1428] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20250831 13:57:56.625136 20692 tablet_bootstrap.cc:492] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.617899 20577 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:56.627836 20691 tablet_bootstrap.cc:492] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Bootstrap starting.
W20250831 13:57:56.627961 20617 leader_election.cc:343] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:56.618371 20467 leader_election.cc:343] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.631981 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:57:56.632790 20699 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.633350 20699 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.633759 20699 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.635676 20574 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:56.636974 20616 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.637892 20697 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.636864 20767 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.638417 20697 raft_consensus.cc:695] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:56.638809 20616 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.640789 20717 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.640669 20697 consensus_queue.cc:237] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.641273 20717 raft_consensus.cc:695] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:56.642037 20717 consensus_queue.cc:237] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.642962 20699 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.640641 20693 ts_tablet_manager.cc:1428] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.234s user 0.014s sys 0.002s
I20250831 13:57:56.627038 20466 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:56.645248 20817 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.644476 20699 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.646340 20693 tablet_bootstrap.cc:492] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.645440 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:56.646271 20817 raft_consensus.cc:695] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:56.649972 20815 consensus_queue.cc:1035] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250831 13:57:56.650974 20817 consensus_queue.cc:237] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.652159 20466 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:57:56.652490 20541 leader_election.cc:343] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.653357 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:56.653857 20502 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.640795 20767 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.653259 20742 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.668933 20742 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.669304 20742 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.669178 20691 tablet_bootstrap.cc:654] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.683504 20767 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:56.686020 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:56.686693 20499 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:56.686738 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.687562 20574 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:56.688033 20617 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:56.688843 20767 raft_consensus.cc:2802] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.689190 20767 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.689364 20693 tablet_bootstrap.cc:654] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.689551 20767 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.691977 20502 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:56.694331 20397 catalog_manager.cc:5582] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:56.696388 20767 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.697599 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:57:56.698669 20699 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.699299 20699 raft_consensus.cc:695] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:56.699687 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:56.700050 20699 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.700410 20502 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.696756 20397 catalog_manager.cc:5582] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:56.712446 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:56.713236 20574 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.696296 20742 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.674448 20818 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.713526 20692 tablet_bootstrap.cc:654] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.720672 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:56.725982 20576 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.724583 20818 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.719646 20400 catalog_manager.cc:5582] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:56.720957 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:56.723881 20693 tablet_bootstrap.cc:492] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.722738 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:56.727265 20400 catalog_manager.cc:5582] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:56.728260 20693 ts_tablet_manager.cc:1397] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.017s
I20250831 13:57:56.710242 20502 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:56.728813 20651 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.727428 20649 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.722107 20767 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:56.722040 20574 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:56.736287 20616 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:56.722378 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:56.722954 20742 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:57:56.737751 20467 leader_election.cc:343] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.737798 20767 raft_consensus.cc:2802] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.738452 20767 raft_consensus.cc:695] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:56.739365 20767 consensus_queue.cc:237] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.743706 20576 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.745229 20467 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:56.746060 20742 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.746532 20742 raft_consensus.cc:695] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:56.747597 20742 consensus_queue.cc:237] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.753657 20649 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.746476 20651 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.757503 20466 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:56.759407 20693 raft_consensus.cc:357] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.760352 20693 raft_consensus.cc:383] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.760726 20693 raft_consensus.cc:738] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.760520 20818 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.761621 20693 consensus_queue.cc:260] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.762666 20818 raft_consensus.cc:695] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:56.771898 20693 ts_tablet_manager.cc:1428] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.042s user 0.007s sys 0.000s
I20250831 13:57:56.763548 20818 consensus_queue.cc:237] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.773854 20693 tablet_bootstrap.cc:492] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.777705 20691 tablet_bootstrap.cc:492] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.778139 20691 ts_tablet_manager.cc:1397] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.150s user 0.007s sys 0.009s
I20250831 13:57:56.779978 20399 catalog_manager.cc:5582] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:56.789629 20693 tablet_bootstrap.cc:654] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.790371 20742 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.793285 20742 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.795109 20742 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.795759 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:56.796352 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.796893 20651 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
W20250831 13:57:56.796957 20467 leader_election.cc:343] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.798062 20466 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:56.798830 20742 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:56.799131 20742 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:56.799381 20742 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.800477 20692 tablet_bootstrap.cc:492] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.800783 20692 ts_tablet_manager.cc:1397] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.001s
I20250831 13:57:56.807613 20692 raft_consensus.cc:357] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.808467 20692 raft_consensus.cc:383] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.808779 20692 raft_consensus.cc:738] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.810706 20742 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.811188 20691 raft_consensus.cc:357] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.812002 20691 raft_consensus.cc:383] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.812318 20691 raft_consensus.cc:738] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.812567 20742 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.812939 20692 consensus_queue.cc:260] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.813382 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:56.813933 20576 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.815341 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:56.816642 20651 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.822942 20576 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.824137 20692 ts_tablet_manager.cc:1428] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20250831 13:57:56.814604 20691 consensus_queue.cc:260] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.826265 20691 ts_tablet_manager.cc:1428] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.048s user 0.006s sys 0.000s
I20250831 13:57:56.827046 20691 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.828027 20692 tablet_bootstrap.cc:492] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.828708 20467 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:56.829521 20818 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:56.830219 20818 raft_consensus.cc:695] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:56.831079 20818 consensus_queue.cc:237] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.833504 20820 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.834144 20820 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.837917 20820 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.839011 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:57:56.840315 20466 leader_election.cc:343] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.842017 20651 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:56.845985 20692 tablet_bootstrap.cc:654] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.846163 20399 catalog_manager.cc:5582] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:56.847734 20691 tablet_bootstrap.cc:654] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.848256 20399 catalog_manager.cc:5582] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:56.851122 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:56.852808 20467 leader_election.cc:343] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:56.853345 20467 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:56.854651 20742 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:56.867859 20691 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.868716 20691 ts_tablet_manager.cc:1397] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.003s
I20250831 13:57:56.871519 20691 raft_consensus.cc:357] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.872025 20693 tablet_bootstrap.cc:492] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.872184 20691 raft_consensus.cc:383] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.872368 20693 ts_tablet_manager.cc:1397] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.099s user 0.033s sys 0.029s
I20250831 13:57:56.872485 20691 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.873452 20691 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.874660 20692 tablet_bootstrap.cc:492] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.875088 20692 ts_tablet_manager.cc:1397] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.011s
I20250831 13:57:56.875300 20693 raft_consensus.cc:357] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.875733 20691 ts_tablet_manager.cc:1428] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 13:57:56.875939 20693 raft_consensus.cc:383] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.876204 20693 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.876633 20691 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.876801 20693 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.877346 20692 raft_consensus.cc:357] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.877916 20692 raft_consensus.cc:383] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.878156 20692 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.879091 20693 ts_tablet_manager.cc:1428] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250831 13:57:56.878805 20692 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.879884 20693 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.880771 20692 ts_tablet_manager.cc:1428] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250831 13:57:56.880491 20400 catalog_manager.cc:5582] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:56.881546 20692 tablet_bootstrap.cc:492] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.882563 20691 tablet_bootstrap.cc:654] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.888733 20693 tablet_bootstrap.cc:654] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.891934 20692 tablet_bootstrap.cc:654] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.895474 20691 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.895574 20692 tablet_bootstrap.cc:492] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.895928 20691 ts_tablet_manager.cc:1397] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20250831 13:57:56.895943 20692 ts_tablet_manager.cc:1397] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20250831 13:57:56.897591 20693 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.898015 20693 ts_tablet_manager.cc:1397] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20250831 13:57:56.898490 20691 raft_consensus.cc:357] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.899169 20691 raft_consensus.cc:383] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.898836 20692 raft_consensus.cc:357] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.899431 20691 raft_consensus.cc:738] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.899632 20692 raft_consensus.cc:383] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.900032 20692 raft_consensus.cc:738] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.900139 20691 consensus_queue.cc:260] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.900812 20692 consensus_queue.cc:260] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.900769 20399 catalog_manager.cc:5582] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:56.902204 20691 ts_tablet_manager.cc:1428] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250831 13:57:56.903542 20692 ts_tablet_manager.cc:1428] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.000s sys 0.005s
I20250831 13:57:56.903728 20691 tablet_bootstrap.cc:492] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.904445 20692 tablet_bootstrap.cc:492] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.912045 20692 tablet_bootstrap.cc:654] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.913815 20691 tablet_bootstrap.cc:654] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.915179 20693 raft_consensus.cc:357] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.915946 20693 raft_consensus.cc:383] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.916251 20693 raft_consensus.cc:738] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.917094 20693 consensus_queue.cc:260] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.920047 20693 ts_tablet_manager.cc:1428] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.022s user 0.006s sys 0.001s
I20250831 13:57:56.921387 20693 tablet_bootstrap.cc:492] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.926438 20693 tablet_bootstrap.cc:654] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.924780 20692 tablet_bootstrap.cc:492] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.926944 20692 ts_tablet_manager.cc:1397] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.007s
I20250831 13:57:56.928979 20691 tablet_bootstrap.cc:492] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.929446 20691 ts_tablet_manager.cc:1397] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.013s
I20250831 13:57:56.930228 20692 raft_consensus.cc:357] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.930909 20692 raft_consensus.cc:383] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.931190 20692 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.932086 20692 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.932509 20691 raft_consensus.cc:357] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.933342 20691 raft_consensus.cc:383] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.933650 20691 raft_consensus.cc:738] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.934525 20692 ts_tablet_manager.cc:1428] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250831 13:57:56.934414 20691 consensus_queue.cc:260] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.935382 20692 tablet_bootstrap.cc:492] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.937036 20691 ts_tablet_manager.cc:1428] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250831 13:57:56.937878 20691 tablet_bootstrap.cc:492] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.940935 20692 tablet_bootstrap.cc:654] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.944378 20691 tablet_bootstrap.cc:654] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.958143 20691 tablet_bootstrap.cc:492] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.958675 20691 ts_tablet_manager.cc:1397] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.007s
I20250831 13:57:56.960431 20699 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:56.960927 20699 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.961162 20691 raft_consensus.cc:357] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.961820 20691 raft_consensus.cc:383] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.962114 20691 raft_consensus.cc:738] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.962919 20699 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:56.962738 20691 consensus_queue.cc:260] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.963812 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:56.964442 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:56.964838 20691 ts_tablet_manager.cc:1428] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.003s sys 0.001s
I20250831 13:57:56.965160 20649 raft_consensus.cc:2391] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:57:56.965461 20692 tablet_bootstrap.cc:492] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:56.965698 20691 tablet_bootstrap.cc:492] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.965986 20692 ts_tablet_manager.cc:1397] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.019s
I20250831 13:57:56.967586 20693 tablet_bootstrap.cc:492] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:56.968001 20693 ts_tablet_manager.cc:1397] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.011s
I20250831 13:57:56.968670 20692 raft_consensus.cc:357] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.969355 20692 raft_consensus.cc:383] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.969676 20692 raft_consensus.cc:738] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.970345 20693 raft_consensus.cc:357] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.970310 20692 consensus_queue.cc:260] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.971122 20693 raft_consensus.cc:383] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.971378 20691 tablet_bootstrap.cc:654] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.971462 20693 raft_consensus.cc:738] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.972175 20693 consensus_queue.cc:260] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.972712 20692 ts_tablet_manager.cc:1428] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:57:56.976455 20692 tablet_bootstrap.cc:492] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:56.979043 20693 ts_tablet_manager.cc:1428] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20250831 13:57:56.980223 20693 tablet_bootstrap.cc:492] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:56.982012 20692 tablet_bootstrap.cc:654] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.970247 20541 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:56.988153 20734 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:56.988258 20693 tablet_bootstrap.cc:654] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:56.989056 20691 tablet_bootstrap.cc:492] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:56.989504 20691 ts_tablet_manager.cc:1397] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.015s
I20250831 13:57:56.991987 20691 raft_consensus.cc:357] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.992705 20691 raft_consensus.cc:383] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:56.993006 20691 raft_consensus.cc:738] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:56.993726 20691 consensus_queue.cc:260] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:56.996304 20691 ts_tablet_manager.cc:1428] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250831 13:57:56.997184 20691 tablet_bootstrap.cc:492] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:56.994601 20734 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:57.000370 20692 tablet_bootstrap.cc:492] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:57.000756 20692 ts_tablet_manager.cc:1397] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20250831 13:57:57.001899 20691 tablet_bootstrap.cc:654] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.003260 20692 raft_consensus.cc:357] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.003935 20692 raft_consensus.cc:383] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.004190 20692 raft_consensus.cc:738] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.004585 20693 tablet_bootstrap.cc:492] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:57.005008 20693 ts_tablet_manager.cc:1397] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.008s
I20250831 13:57:57.004963 20692 consensus_queue.cc:260] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.008160 20692 ts_tablet_manager.cc:1428] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20250831 13:57:57.007987 20693 raft_consensus.cc:357] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.008737 20693 raft_consensus.cc:383] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.009006 20693 raft_consensus.cc:738] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.009253 20692 tablet_bootstrap.cc:492] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:57.009691 20693 consensus_queue.cc:260] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.011828 20693 ts_tablet_manager.cc:1428] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20250831 13:57:57.015770 20693 tablet_bootstrap.cc:492] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:57.016520 20818 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.016896 20818 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.017437 20767 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.018244 20692 tablet_bootstrap.cc:654] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.017966 20767 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.018790 20818 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.020215 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.020905 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.020941 20576 raft_consensus.cc:2391] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:57.022142 20767 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.025017 20693 tablet_bootstrap.cc:654] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
W20250831 13:57:57.024937 20616 leader_election.cc:343] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:57.026438 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.023595 20691 tablet_bootstrap.cc:492] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:57.023926 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.027786 20691 ts_tablet_manager.cc:1397] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.012s
I20250831 13:57:57.029910 20466 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:57.031245 20818 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
W20250831 13:57:57.031704 20617 leader_election.cc:343] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:57.032230 20617 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:57.036123 20720 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:57.036420 20693 tablet_bootstrap.cc:492] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:57.036783 20693 ts_tablet_manager.cc:1397] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.012s
I20250831 13:57:57.038411 20691 raft_consensus.cc:357] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.039112 20691 raft_consensus.cc:383] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.039377 20691 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.040074 20691 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.042117 20691 ts_tablet_manager.cc:1428] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250831 13:57:57.042136 20818 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:57.043081 20693 raft_consensus.cc:357] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.043692 20693 raft_consensus.cc:383] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.043977 20693 raft_consensus.cc:738] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.044514 20693 consensus_queue.cc:260] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.045953 20693 ts_tablet_manager.cc:1428] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250831 13:57:57.046470 20692 tablet_bootstrap.cc:492] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:57.046833 20693 tablet_bootstrap.cc:492] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:57.043712 20691 tablet_bootstrap.cc:492] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:57.046967 20692 ts_tablet_manager.cc:1397] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.000s
I20250831 13:57:57.053350 20692 raft_consensus.cc:357] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.054075 20692 raft_consensus.cc:383] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.054380 20692 raft_consensus.cc:738] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.054787 20691 tablet_bootstrap.cc:654] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.054190 20767 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:57.054960 20770 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:57.055117 20692 consensus_queue.cc:260] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.056910 20692 ts_tablet_manager.cc:1428] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250831 13:57:57.058151 20692 tablet_bootstrap.cc:492] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:57.063091 20693 tablet_bootstrap.cc:654] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.063867 20692 tablet_bootstrap.cc:654] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.071034 20770 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.079883 20717 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.093873 20742 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.103998 20720 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.110427 20720 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.111071 20720 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.111546 20820 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:57.113456 20720 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.115365 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.121893 20734 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:57.134192 20767 consensus_queue.cc:1035] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.136247 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.138290 20617 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:57.143146 20767 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:57.156531 20649 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.170531 20734 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.173362 20717 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.173978 20717 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.175642 20720 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.176174 20720 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.179075 20720 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.180856 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.181687 20574 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:57.180894 20770 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.183099 20770 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.186888 20770 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.187582 20734 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.188234 20734 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.191658 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.192472 20500 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:57.193773 20543 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.194293 20717 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.194741 20699 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.195137 20699 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.195508 20699 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.196326 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.197404 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.197669 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.197798 20500 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:57.197999 20573 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:57.198254 20648 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:57.200990 20466 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.204633 20770 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.205114 20770 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.206512 20734 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.207180 20734 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.207700 20734 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.210529 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.210825 20699 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.211427 20648 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:57.213114 20699 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.199147 20616 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:57.214700 20616 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:57.214818 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:57.215714 20648 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.227128 20720 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.227826 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:57.227722 20720 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.228569 20500 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.243036 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.243803 20574 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:57.223034 20767 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.250280 20767 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.250605 20767 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.223263 20717 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.205531 20770 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.224313 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.259361 20502 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:57.260485 20693 tablet_bootstrap.cc:492] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:57.260984 20693 ts_tablet_manager.cc:1397] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.214s user 0.037s sys 0.050s
I20250831 13:57:57.263726 20742 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.261672 20717 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.275914 20691 tablet_bootstrap.cc:492] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:57.276360 20691 ts_tablet_manager.cc:1397] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.233s user 0.024s sys 0.022s
I20250831 13:57:57.266381 20720 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.277094 20717 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.267330 20742 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.285993 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:57.286147 20448 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:57:57.304394 20693 raft_consensus.cc:357] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.305047 20693 raft_consensus.cc:383] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.305292 20693 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.306008 20693 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.306334 20691 raft_consensus.cc:357] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.307315 20691 raft_consensus.cc:383] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.307593 20691 raft_consensus.cc:738] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.296592 20502 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:57.310967 20691 consensus_queue.cc:260] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.313835 20692 tablet_bootstrap.cc:492] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:57.314168 20692 ts_tablet_manager.cc:1397] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.256s user 0.072s sys 0.112s
I20250831 13:57:57.316159 20869 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.316354 20692 raft_consensus.cc:357] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.316550 20869 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.316972 20692 raft_consensus.cc:383] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.317222 20692 raft_consensus.cc:738] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.317831 20692 consensus_queue.cc:260] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.318328 20869 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.319891 20692 ts_tablet_manager.cc:1428] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:57:57.321478 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.321794 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.322064 20502 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:57.327103 20869 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.327642 20869 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:57.309113 20864 log.cc:927] Time spent T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.051s user 0.000s sys 0.005s
I20250831 13:57:57.331197 20742 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.332068 20734 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.334261 20693 ts_tablet_manager.cc:1428] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.073s user 0.005s sys 0.001s
I20250831 13:57:57.339087 20693 tablet_bootstrap.cc:492] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:57.350901 20692 tablet_bootstrap.cc:492] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:57.363667 20691 ts_tablet_manager.cc:1428] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.087s user 0.006s sys 0.001s
I20250831 13:57:57.370460 20648 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:57.381526 20767 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.365924 20543 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:57:57.369510 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 91059us
I20250831 13:57:57.386741 20869 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.387461 20500 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:57.389115 20871 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:57:57.388528 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 67932.5us
I20250831 13:57:57.390014 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.396304 20871 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.401968 20815 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:57.401911 20872 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.403849 20872 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.404407 20872 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.405014 20815 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:57.405333 20820 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.408842 20697 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.409261 20697 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.417500 20770 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.423400 20577 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.430296 20770 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.431701 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:57.431936 20767 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.432668 20651 raft_consensus.cc:2391] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:57.433399 20697 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.439306 20691 tablet_bootstrap.cc:492] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:57.435201 20871 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.436775 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.448359 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.453768 20500 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:57:57.450544 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.455425 20770 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.455837 20501 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:57.455885 20770 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.464778 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.465520 20770 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.465855 20717 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.467765 20717 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.467367 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.468333 20651 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:57.469575 20466 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.470759 20820 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.471350 20820 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.471763 20820 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.442507 20543 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.478190 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:57.478767 20573 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
W20250831 13:57:57.474584 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44972.2us
I20250831 13:57:57.449872 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.490139 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:57.481246 20871 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:57.491884 20871 raft_consensus.cc:695] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:57.492671 20871 consensus_queue.cc:237] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.486467 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.485359 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.496178 20742 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:57.496517 20818 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 13:57:57.484705 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 43668.8us
I20250831 13:57:57.495630 20717 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250831 13:57:57.488301 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 44292.8us
I20250831 13:57:57.504213 20717 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.505241 20717 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.505100 20742 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:57.501197 20575 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:57.503109 20815 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20250831 13:57:57.507539 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.510941 20692 tablet_bootstrap.cc:654] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.513036 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:57.514742 20502 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.515403 20697 consensus_queue.cc:1035] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.510597 20543 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.522189 20543 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.523249 20699 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.523725 20699 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.524147 20699 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.525777 20693 tablet_bootstrap.cc:654] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.526611 20870 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.522356 20871 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:57:57.527673 20870 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.537747 20870 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.539453 20699 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.533463 20691 tablet_bootstrap.cc:654] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.535267 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.540752 20574 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:57.540849 20871 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.543530 20871 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:57.542151 20699 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.542549 20820 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.549240 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.555416 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:57.553417 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.557047 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.558198 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:57.558805 20499 raft_consensus.cc:2391] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:57:57.558805 20572 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.557235 20500 raft_consensus.cc:2391] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:57:57.555747 20573 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:57.578016 20877 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.579216 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.557235 20651 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:57.588863 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:57.593300 20500 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.593439 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:57.592339 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:57.595124 20498 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.617791 20692 tablet_bootstrap.cc:492] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:57.618234 20692 ts_tablet_manager.cc:1397] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.298s user 0.009s sys 0.008s
I20250831 13:57:57.609018 20870 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:57.598716 20877 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.604565 20467 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:57:57.627856 20500 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
W20250831 13:57:57.628310 20543 leader_election.cc:343] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:57.628510 20877 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.630185 20820 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.631459 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:57.628265 20817 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:57.636579 20691 tablet_bootstrap.cc:492] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:57.640059 20717 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.642149 20498 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:57.644243 20650 raft_consensus.cc:2391] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:57.644120 20543 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.645032 20870 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:57.645572 20870 raft_consensus.cc:695] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:57.646412 20870 consensus_queue.cc:237] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.618605 20572 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:57.649132 20817 raft_consensus.cc:695] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:57.650085 20817 consensus_queue.cc:237] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.675086 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:57.656087 20871 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.676419 20650 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.652959 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.630920 20697 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:57:57.677755 20697 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.679566 20697 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.662855 20691 ts_tablet_manager.cc:1397] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.224s user 0.012s sys 0.011s
I20250831 13:57:57.656740 20872 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.682490 20576 raft_consensus.cc:2391] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 1: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:57:57.669201 20692 raft_consensus.cc:357] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.683970 20692 raft_consensus.cc:383] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.684165 20692 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.684619 20692 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.683035 20650 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:57.686908 20872 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.636955 20569 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.688519 20691 raft_consensus.cc:357] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.690088 20691 raft_consensus.cc:383] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.689733 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:57.690335 20691 raft_consensus.cc:738] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.680131 20717 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:57.691965 20498 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.692888 20871 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.694891 20691 consensus_queue.cc:260] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:57.685663 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 73786.4us
I20250831 13:57:57.652256 20617 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.702047 20692 ts_tablet_manager.cc:1428] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.084s user 0.009s sys 0.005s
I20250831 13:57:57.702819 20692 tablet_bootstrap.cc:492] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:57.708938 20616 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:57.709486 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:57.710050 20502 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:57:57.707516 20877 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:57.710665 20877 raft_consensus.cc:695] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:57.711216 20693 tablet_bootstrap.cc:492] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:57.711640 20693 ts_tablet_manager.cc:1397] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.373s user 0.026s sys 0.016s
I20250831 13:57:57.711437 20877 consensus_queue.cc:237] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.697233 20466 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.713425 20691 ts_tablet_manager.cc:1428] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.033s user 0.002s sys 0.004s
I20250831 13:57:57.714007 20697 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250831 13:57:57.703965 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 67336us
I20250831 13:57:57.714784 20691 tablet_bootstrap.cc:492] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:57.715920 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.716562 20647 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:57.725354 20569 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:57.723089 20693 raft_consensus.cc:357] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:57.744143 20883 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
I20250831 13:57:57.751200 20616 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:57.761270 20693 raft_consensus.cc:383] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.764051 20693 raft_consensus.cc:738] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.766862 20693 consensus_queue.cc:260] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.804441 20692 tablet_bootstrap.cc:654] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.754335 20498 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:57.809306 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:57.810372 20501 raft_consensus.cc:2391] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 1: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:57:57.798038 20697 raft_consensus.cc:2762] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20250831 13:57:57.828931 20617 leader_election.cc:343] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:57.831091 20717 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:57.835005 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:57.838989 20693 ts_tablet_manager.cc:1428] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.127s user 0.010s sys 0.003s
I20250831 13:57:57.839886 20693 tablet_bootstrap.cc:492] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:57.724166 20817 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:57.839004 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:57.845155 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:57.845888 20650 raft_consensus.cc:2391] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:57.842787 20543 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.849881 20870 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:57.844568 20647 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.850333 20870 raft_consensus.cc:695] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:57.852151 20870 consensus_queue.cc:237] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.854514 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:57.856190 20650 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:57:57.857959 20693 tablet_bootstrap.cc:654] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.849457 20817 raft_consensus.cc:695] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:57:57.840754 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 89694.8us
I20250831 13:57:57.843196 20502 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:57:57.863495 20397 catalog_manager.cc:5582] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:57.867482 20543 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.869885 20871 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:57:57.870703 20871 raft_consensus.cc:695] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:57.871448 20871 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.871798 20650 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:57:57.859042 20817 consensus_queue.cc:237] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.910813 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.951694 20820 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:57.893309 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:57.956358 20820 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.959648 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.971993 20691 tablet_bootstrap.cc:654] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:57.972183 20820 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.973062 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:57.962869 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:57.973443 20570 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:57.973587 20650 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:57.974591 20466 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:57.962999 20647 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:57.975436 20692 tablet_bootstrap.cc:492] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:57.975963 20692 ts_tablet_manager.cc:1397] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.273s user 0.015s sys 0.002s
I20250831 13:57:57.974804 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:57.977355 20817 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.977417 20572 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:57.977792 20817 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.978171 20817 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.980202 20871 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:57.983116 20692 raft_consensus.cc:357] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.987825 20693 tablet_bootstrap.cc:492] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:57.989178 20871 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:57.989271 20692 raft_consensus.cc:383] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.989714 20693 ts_tablet_manager.cc:1397] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.150s user 0.017s sys 0.005s
I20250831 13:57:57.992348 20693 raft_consensus.cc:357] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.992964 20693 raft_consensus.cc:383] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:57.993258 20693 raft_consensus.cc:738] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.988493 20817 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.994910 20693 consensus_queue.cc:260] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.996562 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:57.989683 20692 raft_consensus.cc:738] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:57.998297 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:57.999022 20647 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:57.998008 20817 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:57.999949 20692 consensus_queue.cc:260] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:57.989629 20871 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.003362 20692 ts_tablet_manager.cc:1428] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250831 13:57:58.004838 20692 tablet_bootstrap.cc:492] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:58.003834 20693 ts_tablet_manager.cc:1428] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250831 13:57:58.006385 20872 consensus_queue.cc:1035] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.006752 20693 tablet_bootstrap.cc:492] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:58.008379 20647 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:58.010463 20466 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:58.011471 20820 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:58.011962 20820 raft_consensus.cc:695] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:58.012964 20820 consensus_queue.cc:237] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.015465 20616 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.017117 20697 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 13:57:58.018008 20871 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.019213 20572 raft_consensus.cc:2391] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 1: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:57:58.019487 20691 tablet_bootstrap.cc:492] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:58.019920 20691 ts_tablet_manager.cc:1397] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.306s user 0.063s sys 0.074s
I20250831 13:57:58.020221 20871 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.021404 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:58.021862 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:58.022706 20647 raft_consensus.cc:2391] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:57:58.022792 20692 tablet_bootstrap.cc:654] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.026527 20693 tablet_bootstrap.cc:654] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.023991 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.024317 20616 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.029788 20697 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:57:58.030282 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.033036 20691 raft_consensus.cc:357] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.033818 20691 raft_consensus.cc:383] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.034147 20691 raft_consensus.cc:738] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.034960 20691 consensus_queue.cc:260] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.034515 20877 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.037920 20877 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.038235 20877 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.039991 20691 ts_tablet_manager.cc:1428] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.020s user 0.004s sys 0.004s
I20250831 13:57:58.040889 20691 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:58.034724 20871 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 13:57:58.080008 20870 consensus_queue.cc:1035] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.080742 20693 tablet_bootstrap.cc:492] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:58.081182 20693 ts_tablet_manager.cc:1397] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.075s user 0.014s sys 0.007s
I20250831 13:57:58.083639 20693 raft_consensus.cc:357] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.083957 20877 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.084297 20693 raft_consensus.cc:383] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.084996 20396 catalog_manager.cc:5582] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:58.095170 20693 raft_consensus.cc:738] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.096923 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:58.097479 20570 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.085832 20877 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.089190 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:58.086964 20691 tablet_bootstrap.cc:654] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.104075 20570 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:58.105298 20870 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.105409 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:58.105743 20870 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.096864 20693 consensus_queue.cc:260] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.107615 20870 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.108042 20693 ts_tablet_manager.cc:1428] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.027s user 0.003s sys 0.004s
I20250831 13:57:58.108668 20697 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:58.108903 20693 tablet_bootstrap.cc:492] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:58.109138 20697 raft_consensus.cc:695] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:58.113157 20697 consensus_queue.cc:237] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:58.116189 20617 leader_election.cc:343] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.116989 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:58.116528 20400 catalog_manager.cc:5582] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:58.117543 20647 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:58.120615 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.118254 20400 catalog_manager.cc:5582] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:58.121739 20502 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:58.123524 20541 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:58.125164 20870 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.125496 20870 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.125837 20870 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.126451 20400 catalog_manager.cc:5582] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:58.129732 20692 tablet_bootstrap.cc:492] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:58.130165 20692 ts_tablet_manager.cc:1397] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.126s user 0.016s sys 0.004s
I20250831 13:57:58.133908 20693 tablet_bootstrap.cc:654] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.142827 20871 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.144608 20692 raft_consensus.cc:357] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.144768 20870 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.145282 20692 raft_consensus.cc:383] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.145546 20692 raft_consensus.cc:738] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.146535 20870 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.146189 20692 consensus_queue.cc:260] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.150183 20817 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.150358 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:58.150970 20650 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.150669 20817 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.153964 20817 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.155606 20691 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:58.156046 20691 ts_tablet_manager.cc:1397] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.115s user 0.015s sys 0.009s
I20250831 13:57:58.156386 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:58.158047 20502 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.160729 20397 catalog_manager.cc:5582] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:58.163920 20397 catalog_manager.cc:5582] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:58.163120 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:58.167117 20692 ts_tablet_manager.cc:1428] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.037s user 0.007s sys 0.000s
I20250831 13:57:58.167946 20692 tablet_bootstrap.cc:492] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
W20250831 13:57:58.168432 20466 leader_election.cc:343] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.169154 20693 tablet_bootstrap.cc:492] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:58.169116 20691 raft_consensus.cc:357] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.169672 20693 ts_tablet_manager.cc:1397] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.000s
I20250831 13:57:58.169844 20691 raft_consensus.cc:383] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.170171 20691 raft_consensus.cc:738] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.170790 20691 consensus_queue.cc:260] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.171732 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.172466 20650 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:58.173494 20541 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:58.157397 20697 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.174973 20697 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.175437 20691 ts_tablet_manager.cc:1428] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250831 13:57:58.176298 20691 tablet_bootstrap.cc:492] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:58.177054 20870 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:58.177547 20870 raft_consensus.cc:695] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:57:58.178035 20467 leader_election.cc:343] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.178308 20870 consensus_queue.cc:237] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.182096 20467 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:58.179559 20502 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:58.179680 20693 raft_consensus.cc:357] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.184407 20817 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:58.179307 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.183902 20692 tablet_bootstrap.cc:654] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.185423 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.186228 20820 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 13:57:58.186740 20616 leader_election.cc:343] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.184506 20693 raft_consensus.cc:383] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250831 13:57:58.187337 20617 leader_election.cc:343] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.187536 20693 raft_consensus.cc:738] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.187739 20617 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.188156 20693 consensus_queue.cc:260] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.188510 20767 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:58.183152 20697 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.191031 20693 ts_tablet_manager.cc:1428] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20250831 13:57:58.191869 20693 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:58.192775 20691 tablet_bootstrap.cc:654] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.197575 20693 tablet_bootstrap.cc:654] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.203909 20691 tablet_bootstrap.cc:492] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:58.204327 20691 ts_tablet_manager.cc:1397] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.006s
I20250831 13:57:58.204074 20400 catalog_manager.cc:5582] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:58.206485 20692 tablet_bootstrap.cc:492] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:58.207150 20691 raft_consensus.cc:357] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.207623 20692 ts_tablet_manager.cc:1397] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.004s
I20250831 13:57:58.207823 20691 raft_consensus.cc:383] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.208117 20691 raft_consensus.cc:738] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.208832 20691 consensus_queue.cc:260] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.209942 20692 raft_consensus.cc:357] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.210873 20691 ts_tablet_manager.cc:1428] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250831 13:57:58.211681 20691 tablet_bootstrap.cc:492] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:58.213457 20817 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.215076 20396 catalog_manager.cc:5582] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:58.217918 20691 tablet_bootstrap.cc:654] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.221865 20872 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.229871 20691 tablet_bootstrap.cc:492] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:58.230268 20691 ts_tablet_manager.cc:1397] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20250831 13:57:58.232472 20691 raft_consensus.cc:357] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.233071 20691 raft_consensus.cc:383] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.210601 20692 raft_consensus.cc:383] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.233435 20691 raft_consensus.cc:738] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.233702 20692 raft_consensus.cc:738] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.234305 20691 consensus_queue.cc:260] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.236361 20692 consensus_queue.cc:260] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.241040 20692 ts_tablet_manager.cc:1428] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20250831 13:57:58.241652 20692 tablet_bootstrap.cc:492] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:58.248391 20692 tablet_bootstrap.cc:654] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.237301 20691 ts_tablet_manager.cc:1428] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20250831 13:57:58.251185 20691 tablet_bootstrap.cc:492] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:58.257644 20870 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:58.259016 20691 tablet_bootstrap.cc:654] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.261302 20693 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:58.261714 20693 ts_tablet_manager.cc:1397] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.012s
I20250831 13:57:58.263167 20397 catalog_manager.cc:5582] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:58.264225 20693 raft_consensus.cc:357] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.264850 20693 raft_consensus.cc:383] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.265105 20693 raft_consensus.cc:738] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.265661 20693 consensus_queue.cc:260] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.269482 20697 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.270215 20693 ts_tablet_manager.cc:1428] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250831 13:57:58.270072 20697 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.274057 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.274609 20499 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:58.276409 20617 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:58.277465 20767 raft_consensus.cc:2802] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.277802 20767 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.278244 20767 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.284965 20697 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.285734 20767 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.286372 20693 tablet_bootstrap.cc:492] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:58.286150 20692 tablet_bootstrap.cc:492] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:58.289434 20692 ts_tablet_manager.cc:1397] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.018s
I20250831 13:57:58.292086 20693 tablet_bootstrap.cc:654] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.293534 20692 raft_consensus.cc:357] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.294198 20692 raft_consensus.cc:383] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.294515 20692 raft_consensus.cc:738] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.295285 20692 consensus_queue.cc:260] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.296234 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:58.296813 20499 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.297467 20691 tablet_bootstrap.cc:492] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:58.297441 20692 ts_tablet_manager.cc:1428] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250831 13:57:58.297989 20691 ts_tablet_manager.cc:1397] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.010s
I20250831 13:57:58.298514 20692 tablet_bootstrap.cc:492] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:58.300251 20691 raft_consensus.cc:357] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.300860 20691 raft_consensus.cc:383] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.301113 20691 raft_consensus.cc:738] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.301710 20691 consensus_queue.cc:260] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.303925 20691 ts_tablet_manager.cc:1428] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:57:58.305262 20692 tablet_bootstrap.cc:654] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.306488 20691 tablet_bootstrap.cc:492] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:58.309243 20499 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:58.289345 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.311342 20617 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:58.312665 20697 raft_consensus.cc:2802] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:58.313302 20697 raft_consensus.cc:695] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:57:58.313468 20616 leader_election.cc:343] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.314509 20693 tablet_bootstrap.cc:492] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:58.314074 20697 consensus_queue.cc:237] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.315013 20693 ts_tablet_manager.cc:1397] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.001s
I20250831 13:57:58.321738 20693 raft_consensus.cc:357] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.322419 20693 raft_consensus.cc:383] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.322732 20693 raft_consensus.cc:738] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.323370 20693 consensus_queue.cc:260] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.325469 20693 ts_tablet_manager.cc:1428] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250831 13:57:58.328926 20691 tablet_bootstrap.cc:654] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.331435 20693 tablet_bootstrap.cc:492] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:58.338968 20872 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:58.340996 20767 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.341988 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:58.342571 20692 tablet_bootstrap.cc:492] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:58.343073 20692 ts_tablet_manager.cc:1397] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.022s
W20250831 13:57:58.343230 20616 leader_election.cc:343] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.345531 20692 raft_consensus.cc:357] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.346194 20693 tablet_bootstrap.cc:654] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.346164 20692 raft_consensus.cc:383] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.346575 20692 raft_consensus.cc:738] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.347191 20692 consensus_queue.cc:260] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.349364 20817 consensus_queue.cc:1035] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.349898 20692 ts_tablet_manager.cc:1428] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:57:58.350778 20692 tablet_bootstrap.cc:492] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:58.347165 20697 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.358119 20697 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.360220 20400 catalog_manager.cc:5582] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:58.363451 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.364181 20692 tablet_bootstrap.cc:654] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.364233 20502 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:58.379529 20617 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:58.383841 20767 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.384268 20767 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.384641 20767 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.389492 20717 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:58.393739 20650 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:57:58.395320 20872 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.398000 20697 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.400159 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.400688 20572 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:58.405207 20872 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.415221 20872 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.425808 20691 tablet_bootstrap.cc:492] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:58.426258 20691 ts_tablet_manager.cc:1397] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.120s user 0.025s sys 0.018s
I20250831 13:57:58.426573 20697 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250831 13:57:58.434782 20872 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.435463 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.454802 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:58.455293 20817 consensus_queue.cc:1035] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20250831 13:57:58.473410 20693 tablet_bootstrap.cc:492] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:58.473894 20693 ts_tablet_manager.cc:1397] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.143s user 0.028s sys 0.034s
I20250831 13:57:58.486045 20691 raft_consensus.cc:357] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.486738 20691 raft_consensus.cc:383] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.486979 20691 raft_consensus.cc:738] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.487416 20691 consensus_queue.cc:260] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.488930 20691 ts_tablet_manager.cc:1428] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.062s user 0.005s sys 0.001s
I20250831 13:57:58.498522 20691 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: Bootstrap starting.
W20250831 13:57:58.475777 20543 leader_election.cc:343] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.501987 20693 raft_consensus.cc:357] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.502743 20693 raft_consensus.cc:383] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.503007 20693 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.503559 20693 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:58.503991 20541 leader_election.cc:343] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.482441 20767 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.511059 20767 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.517572 20541 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.523177 20872 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:58.543886 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:58.546727 20501 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.564044 20693 ts_tablet_manager.cc:1428] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.090s user 0.006s sys 0.001s
I20250831 13:57:58.565423 20693 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:58.569064 20692 tablet_bootstrap.cc:492] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:58.569670 20692 ts_tablet_manager.cc:1397] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.219s user 0.012s sys 0.037s
W20250831 13:57:58.574774 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 92347.8us
W20250831 13:57:58.580035 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 60165.9us
I20250831 13:57:58.581176 20820 consensus_queue.cc:1035] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.588342 20691 tablet_bootstrap.cc:654] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.582496 20692 raft_consensus.cc:357] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.589653 20692 raft_consensus.cc:383] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.590013 20692 raft_consensus.cc:738] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.590806 20692 consensus_queue.cc:260] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:58.580691 20448 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:57:58.597605 20817 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.598153 20817 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.600169 20817 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.601176 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:58.601822 20650 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:58.602617 20692 ts_tablet_manager.cc:1428] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.033s user 0.009s sys 0.000s
I20250831 13:57:58.603849 20692 tablet_bootstrap.cc:492] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:58.609871 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.602042 20935 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.610008 20499 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:57:58.605787 20871 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.614941 20871 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:58.620460 20467 leader_election.cc:343] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.615770 20935 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.624609 20935 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.625211 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.626415 20577 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:58.627236 20871 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.609867 20501 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:58.635629 20617 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:58.636106 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.636445 20767 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:58.636682 20498 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:58.636942 20767 raft_consensus.cc:695] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:58.637185 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:58.628096 20870 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20250831 13:57:58.639237 20543 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:58.639382 20870 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.640002 20872 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.640374 20872 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.637764 20767 consensus_queue.cc:237] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.640707 20872 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.637774 20570 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.651119 20693 tablet_bootstrap.cc:654] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.651657 20877 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.654009 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:58.654608 20651 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:58.665287 20692 tablet_bootstrap.cc:654] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.669334 20466 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:57:58.670238 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 49506.1us
I20250831 13:57:58.672039 20817 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.672397 20817 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.672466 20570 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:58.672751 20817 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.652480 20467 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:58.676400 20935 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.676764 20935 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.677134 20935 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.677723 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:58.678328 20651 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:58.680335 20872 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.682142 20820 consensus_queue.cc:1035] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.695554 20691 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:58.696105 20691 ts_tablet_manager.cc:1397] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.198s user 0.012s sys 0.006s
I20250831 13:57:58.702131 20693 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:58.717190 20693 ts_tablet_manager.cc:1397] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.152s user 0.012s sys 0.011s
I20250831 13:57:58.718968 20691 raft_consensus.cc:357] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.719049 20871 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:58.720010 20691 raft_consensus.cc:383] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.720314 20691 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.721123 20872 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.721614 20691 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.732945 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:58.733494 20651 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.727432 20501 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.731573 20871 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:57:58.748553 20616 consensus_peers.cc:489] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 -> Peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Couldn't send request to peer 212b66976f874aa682cae0fb77928786. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:57:58.749393 20693 raft_consensus.cc:357] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.749995 20693 raft_consensus.cc:383] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.750250 20693 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.746407 20871 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.750795 20693 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.752503 20871 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.752962 20935 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.754026 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:58.755600 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.770838 20691 ts_tablet_manager.cc:1428] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.074s user 0.010s sys 0.000s
I20250831 13:57:58.772058 20691 tablet_bootstrap.cc:492] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:58.754638 20649 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:58.785284 20541 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:58.786563 20872 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.786931 20872 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.787245 20872 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.761178 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:58.789651 20693 ts_tablet_manager.cc:1428] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.072s user 0.005s sys 0.001s
I20250831 13:57:58.761430 20935 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.791378 20693 tablet_bootstrap.cc:492] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:58.791862 20499 raft_consensus.cc:2391] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:57:58.787869 20543 leader_election.cc:343] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.787160 20880 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.800029 20817 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.801250 20934 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.792886 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:58.804679 20692 tablet_bootstrap.cc:492] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:58.804982 20817 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.804714 20934 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.807185 20691 tablet_bootstrap.cc:654] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.808979 20692 ts_tablet_manager.cc:1397] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.205s user 0.022s sys 0.015s
I20250831 13:57:58.807407 20717 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.795847 20935 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.811874 20717 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.816682 20717 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.819417 20577 raft_consensus.cc:2391] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 1: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:57:58.819783 20934 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.813459 20935 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.821583 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:58.823179 20617 leader_election.cc:343] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.821692 20935 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.823539 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:58.829020 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.811724 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:58.829591 20498 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:58.831625 20617 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:58.833971 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.834544 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.834417 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:58.835577 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:58.836191 20467 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:58.836983 20877 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:58.818902 20651 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:58.837765 20466 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:58.830726 20692 raft_consensus.cc:357] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.841627 20692 raft_consensus.cc:383] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.841980 20692 raft_consensus.cc:738] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.842427 20692 consensus_queue.cc:260] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.843338 20693 tablet_bootstrap.cc:654] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.844635 20541 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:57:58.845196 20616 leader_election.cc:343] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.846505 20872 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.846868 20817 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250831 13:57:58.846208 20616 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.845947 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:58.839900 20877 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:58.848500 20872 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.845638 20935 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:57:58.850620 20877 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.851404 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:58.851931 20648 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
W20250831 13:57:58.857980 20616 leader_election.cc:343] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.859213 20717 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:58.861415 20692 ts_tablet_manager.cc:1428] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20250831 13:57:58.861886 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:57:58.861080 20467 leader_election.cc:343] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.866636 20648 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:58.845716 20650 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.866153 20953 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
W20250831 13:57:58.874405 20543 leader_election.cc:343] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.872113 20953 raft_consensus.cc:695] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:58.881711 20692 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:58.885551 20767 consensus_queue.cc:1035] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:58.886277 20877 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.890750 20953 consensus_queue.cc:237] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.901639 20541 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.904911 20957 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:58.917963 20691 tablet_bootstrap.cc:492] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:58.918354 20691 ts_tablet_manager.cc:1397] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.147s user 0.025s sys 0.020s
I20250831 13:57:58.927237 20691 raft_consensus.cc:357] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.928149 20691 raft_consensus.cc:383] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.928470 20691 raft_consensus.cc:738] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.929220 20691 consensus_queue.cc:260] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.931914 20691 ts_tablet_manager.cc:1428] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250831 13:57:58.932883 20691 tablet_bootstrap.cc:492] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:58.917874 20957 raft_consensus.cc:695] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:58.942901 20957 consensus_queue.cc:237] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.947666 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:58.947794 20869 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:58.947906 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:58.948292 20498 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:58.948310 20869 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:58.949168 20616 leader_election.cc:343] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.950198 20869 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:58.950878 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:58.944964 20877 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:58.945039 20650 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:58.952733 20692 tablet_bootstrap.cc:654] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.955960 20466 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:57:58.954001 20543 leader_election.cc:343] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.956971 20935 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:58.957419 20935 raft_consensus.cc:695] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:58.958186 20935 consensus_queue.cc:237] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.968080 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:57:58.969916 20541 leader_election.cc:343] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:58.970391 20541 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:58.972667 20691 tablet_bootstrap.cc:654] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:58.975076 20693 tablet_bootstrap.cc:492] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:58.975476 20693 ts_tablet_manager.cc:1397] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.184s user 0.020s sys 0.015s
I20250831 13:57:58.977545 20498 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:58.979012 20617 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:58.981839 20693 raft_consensus.cc:357] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.982340 20693 raft_consensus.cc:383] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:58.982501 20693 raft_consensus.cc:738] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:58.983004 20693 consensus_queue.cc:260] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.984256 20957 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:58.985095 20697 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:58.985656 20697 raft_consensus.cc:695] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:58.986733 20697 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:58.995108 20693 ts_tablet_manager.cc:1428] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250831 13:57:59.012616 20693 tablet_bootstrap.cc:492] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.020097 20693 tablet_bootstrap.cc:654] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.027352 20400 catalog_manager.cc:5582] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:59.032420 20693 tablet_bootstrap.cc:492] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.033100 20693 ts_tablet_manager.cc:1397] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20250831 13:57:59.036137 20953 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.036759 20953 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.037060 20693 raft_consensus.cc:357] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.038228 20693 raft_consensus.cc:383] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.038618 20693 raft_consensus.cc:738] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.039603 20693 consensus_queue.cc:260] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.040352 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:59.040953 20498 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:59.041390 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.041940 20648 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 0.
I20250831 13:57:59.042259 20543 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:59.043011 20957 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:59.043215 20693 ts_tablet_manager.cc:1428] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20250831 13:57:59.043370 20957 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:59.043834 20957 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.044443 20693 tablet_bootstrap.cc:492] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.045645 20692 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:59.046061 20692 ts_tablet_manager.cc:1397] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.165s user 0.027s sys 0.037s
I20250831 13:57:59.043254 20953 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.048787 20692 raft_consensus.cc:357] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.049612 20692 raft_consensus.cc:383] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.049940 20692 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.050814 20693 tablet_bootstrap.cc:654] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.050556 20957 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.050590 20692 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.053388 20692 ts_tablet_manager.cc:1428] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:57:59.053299 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:59.054039 20498 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.054328 20692 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:59.062567 20693 tablet_bootstrap.cc:492] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.062990 20957 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.063218 20693 ts_tablet_manager.cc:1397] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20250831 13:57:59.065012 20692 tablet_bootstrap.cc:654] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.065248 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:59.065852 20650 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.066299 20693 raft_consensus.cc:357] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.067028 20693 raft_consensus.cc:383] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.067332 20693 raft_consensus.cc:738] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.068042 20693 consensus_queue.cc:260] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.070252 20693 ts_tablet_manager.cc:1428] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:57:59.071154 20693 tablet_bootstrap.cc:492] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.072082 20650 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:59.073074 20541 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:59.062579 20498 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:57:59.077051 20957 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:59.077358 20693 tablet_bootstrap.cc:654] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.077870 20957 raft_consensus.cc:695] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:57:59.078774 20957 consensus_queue.cc:237] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.075736 20691 tablet_bootstrap.cc:492] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.080746 20691 ts_tablet_manager.cc:1397] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.148s user 0.024s sys 0.042s
I20250831 13:57:59.081544 20692 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:59.082134 20692 ts_tablet_manager.cc:1397] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.000s
I20250831 13:57:59.083945 20691 raft_consensus.cc:357] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.084766 20691 raft_consensus.cc:383] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.085101 20691 raft_consensus.cc:738] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.087731 20691 consensus_queue.cc:260] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.089061 20693 tablet_bootstrap.cc:492] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.089514 20693 ts_tablet_manager.cc:1397] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20250831 13:57:59.089908 20691 ts_tablet_manager.cc:1428] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250831 13:57:59.089198 20692 raft_consensus.cc:357] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.090806 20691 tablet_bootstrap.cc:492] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.090778 20692 raft_consensus.cc:383] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.091634 20692 raft_consensus.cc:738] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.092255 20693 raft_consensus.cc:357] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.092947 20693 raft_consensus.cc:383] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.093245 20693 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.093994 20693 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.096513 20691 tablet_bootstrap.cc:654] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.096534 20693 ts_tablet_manager.cc:1428] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250831 13:57:59.097635 20693 tablet_bootstrap.cc:492] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.099754 20877 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.092284 20692 consensus_queue.cc:260] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.102396 20818 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.102862 20818 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.104779 20818 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.106856 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.107425 20650 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:59.107671 20693 tablet_bootstrap.cc:654] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.108434 20466 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:59.109398 20818 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:59.109735 20818 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:59.110098 20818 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.110590 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.111208 20575 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:59.112381 20397 catalog_manager.cc:5582] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:59.115245 20691 tablet_bootstrap.cc:492] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.116071 20691 ts_tablet_manager.cc:1397] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.006s
I20250831 13:57:59.117452 20818 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.117868 20693 tablet_bootstrap.cc:492] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.118350 20693 ts_tablet_manager.cc:1397] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.002s
I20250831 13:57:59.119812 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:59.120319 20575 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.120560 20693 raft_consensus.cc:357] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.121189 20693 raft_consensus.cc:383] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.121454 20693 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.122087 20693 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.122393 20691 raft_consensus.cc:357] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.123154 20691 raft_consensus.cc:383] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.123446 20691 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.124250 20693 ts_tablet_manager.cc:1428] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:57:59.124178 20691 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.125116 20693 tablet_bootstrap.cc:492] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.126582 20691 ts_tablet_manager.cc:1428] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.010s user 0.003s sys 0.002s
I20250831 13:57:59.127480 20691 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.127877 20692 ts_tablet_manager.cc:1428] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.045s user 0.005s sys 0.004s
I20250831 13:57:59.127206 20575 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:59.133016 20693 tablet_bootstrap.cc:654] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.133757 20467 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:59.134896 20935 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:59.135442 20935 raft_consensus.cc:695] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:59.129237 20692 tablet_bootstrap.cc:492] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:59.134830 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:59.137334 20650 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.139707 20697 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.136296 20935 consensus_queue.cc:237] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.146067 20939 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.133077 20691 tablet_bootstrap.cc:654] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.146476 20939 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.148290 20939 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.148898 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:59.149989 20467 leader_election.cc:343] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.151271 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:57:59.152572 20466 leader_election.cc:343] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.152987 20466 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:59.153970 20939 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.158366 20650 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:59.163853 20396 catalog_manager.cc:5582] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:59.165343 20396 catalog_manager.cc:5582] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:59.166002 20691 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.166566 20691 ts_tablet_manager.cc:1397] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.008s
I20250831 13:57:59.169544 20691 raft_consensus.cc:357] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.170389 20691 raft_consensus.cc:383] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.170740 20691 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.171609 20691 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.174418 20691 ts_tablet_manager.cc:1428] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:57:59.175328 20691 tablet_bootstrap.cc:492] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.181663 20691 tablet_bootstrap.cc:654] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.171612 20818 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.186571 20692 tablet_bootstrap.cc:654] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.187557 20818 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.188012 20818 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.189888 20818 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.191617 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.191622 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.193840 20648 raft_consensus.cc:2391] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 1: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:57:59.195108 20693 tablet_bootstrap.cc:492] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.195523 20693 ts_tablet_manager.cc:1397] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.071s user 0.018s sys 0.031s
I20250831 13:57:59.197366 20467 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:59.198279 20818 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.198349 20693 raft_consensus.cc:357] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.199055 20693 raft_consensus.cc:383] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.199347 20693 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.200685 20957 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.201297 20957 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.203013 20693 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.204667 20818 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.205000 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:59.205152 20693 ts_tablet_manager.cc:1428] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250831 13:57:59.206955 20693 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.209410 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.210405 20648 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:57:59.212692 20692 tablet_bootstrap.cc:492] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:59.213126 20692 ts_tablet_manager.cc:1397] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.004s
I20250831 13:57:59.213655 20541 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:59.213171 20693 tablet_bootstrap.cc:654] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.214298 20953 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.215682 20692 raft_consensus.cc:357] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.216316 20692 raft_consensus.cc:383] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.216593 20692 raft_consensus.cc:738] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.217344 20692 consensus_queue.cc:260] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.220037 20398 catalog_manager.cc:5582] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:59.205556 20957 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.225793 20692 ts_tablet_manager.cc:1428] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250831 13:57:59.225855 20953 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.227766 20691 tablet_bootstrap.cc:492] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.234860 20691 ts_tablet_manager.cc:1397] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.060s user 0.011s sys 0.019s
I20250831 13:57:59.236547 20692 tablet_bootstrap.cc:492] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:59.248116 20691 raft_consensus.cc:357] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.248608 20691 raft_consensus.cc:383] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.248806 20691 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.249246 20691 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.252631 20691 ts_tablet_manager.cc:1428] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
I20250831 13:57:59.253227 20691 tablet_bootstrap.cc:492] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.257690 20935 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.258193 20691 tablet_bootstrap.cc:654] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.258150 20935 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.262935 20935 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.263984 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:59.267551 20467 leader_election.cc:343] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.268827 20692 tablet_bootstrap.cc:654] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.269965 20693 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.270534 20693 ts_tablet_manager.cc:1397] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.064s user 0.030s sys 0.014s
I20250831 13:57:59.272373 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.273005 20650 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:59.283279 20466 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:59.284307 20570 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.284957 20817 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:59.285382 20817 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:59.285759 20817 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.294510 20693 raft_consensus.cc:357] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.296121 20693 raft_consensus.cc:383] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.296469 20693 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.297200 20693 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.302068 20693 ts_tablet_manager.cc:1428] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20250831 13:57:59.304708 20693 tablet_bootstrap.cc:492] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.311117 20817 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.304280 20398 catalog_manager.cc:5582] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:59.314280 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:59.314929 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:59.309742 20691 tablet_bootstrap.cc:492] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.315464 20650 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.315645 20691 ts_tablet_manager.cc:1397] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.030s
I20250831 13:57:59.317070 20693 tablet_bootstrap.cc:654] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.317916 20877 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:57:59.318462 20467 leader_election.cc:343] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.318576 20877 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.319434 20817 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.320431 20877 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.320928 20691 raft_consensus.cc:357] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.321560 20691 raft_consensus.cc:383] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.321844 20691 raft_consensus.cc:738] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.322453 20691 consensus_queue.cc:260] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.322765 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:59.323402 20497 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:59.323908 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.324728 20691 ts_tablet_manager.cc:1428] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250831 13:57:59.324609 20617 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:57:59.325156 20616 leader_election.cc:343] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.325752 20697 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:59.326134 20697 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:59.326458 20697 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.324949 20650 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:59.327862 20466 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:59.328950 20935 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:59.329397 20935 raft_consensus.cc:695] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:59.330170 20935 consensus_queue.cc:237] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.333659 20697 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.338994 20697 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.339248 20396 catalog_manager.cc:5582] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:59.340293 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:59.340334 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:59.340965 20497 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.341817 20691 tablet_bootstrap.cc:492] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.339156 20692 tablet_bootstrap.cc:492] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:59.343389 20692 ts_tablet_manager.cc:1397] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.107s user 0.009s sys 0.007s
W20250831 13:57:59.341723 20616 leader_election.cc:343] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.345963 20692 raft_consensus.cc:357] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.350828 20692 raft_consensus.cc:383] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.351173 20692 raft_consensus.cc:738] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.352026 20692 consensus_queue.cc:260] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.348891 20691 tablet_bootstrap.cc:654] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.354087 20497 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:59.356886 20617 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:59.360091 20717 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:59.360555 20717 raft_consensus.cc:695] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:59.361366 20693 tablet_bootstrap.cc:492] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.361352 20717 consensus_queue.cc:237] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.362076 20693 ts_tablet_manager.cc:1397] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.058s user 0.011s sys 0.002s
I20250831 13:57:59.362828 20957 consensus_queue.cc:1035] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.363171 20692 ts_tablet_manager.cc:1428] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.019s user 0.004s sys 0.001s
I20250831 13:57:59.364239 20692 tablet_bootstrap.cc:492] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:59.365384 20693 raft_consensus.cc:357] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.366214 20693 raft_consensus.cc:383] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.366544 20693 raft_consensus.cc:738] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.367408 20693 consensus_queue.cc:260] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.369122 20398 catalog_manager.cc:5582] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:57:59.370507 20693 ts_tablet_manager.cc:1428] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20250831 13:57:59.371766 20693 tablet_bootstrap.cc:492] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.381212 20692 tablet_bootstrap.cc:654] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.381788 20693 tablet_bootstrap.cc:654] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.384799 20869 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.385262 20869 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.387152 20869 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.392046 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:59.392158 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.392788 20498 raft_consensus.cc:2391] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:59.394057 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:59.395107 20954 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.414579 20953 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.415248 20953 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.415624 20398 catalog_manager.cc:5582] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:59.403477 20957 consensus_queue.cc:1035] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20250831 13:57:59.419256 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.422374 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:59.424103 20541 leader_election.cc:343] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:59.425330 20543 leader_election.cc:343] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.425942 20543 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:59.427029 20953 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.432430 20870 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.438617 20691 tablet_bootstrap.cc:492] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.439285 20691 ts_tablet_manager.cc:1397] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.114s user 0.024s sys 0.037s
I20250831 13:57:59.453284 20954 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.463215 20693 tablet_bootstrap.cc:492] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.464359 20692 tablet_bootstrap.cc:492] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:59.467582 20693 ts_tablet_manager.cc:1397] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.096s user 0.021s sys 0.016s
I20250831 13:57:59.467381 20954 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.467804 20692 ts_tablet_manager.cc:1397] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.104s user 0.031s sys 0.037s
I20250831 13:57:59.463479 20935 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250831 13:57:59.471412 20692 raft_consensus.cc:357] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.472051 20692 raft_consensus.cc:383] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.472340 20692 raft_consensus.cc:738] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.472364 20817 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250831 13:57:59.477681 20692 consensus_queue.cc:260] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.479920 20692 ts_tablet_manager.cc:1428] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250831 13:57:59.480831 20692 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:59.460584 20691 raft_consensus.cc:357] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.484234 20691 raft_consensus.cc:383] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.484004 20693 raft_consensus.cc:357] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.484594 20691 raft_consensus.cc:738] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.484899 20693 raft_consensus.cc:383] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.485239 20693 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.485457 20691 consensus_queue.cc:260] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.485908 20693 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.489131 20693 ts_tablet_manager.cc:1428] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250831 13:57:59.490211 20693 tablet_bootstrap.cc:492] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.490204 20691 ts_tablet_manager.cc:1428] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.051s user 0.005s sys 0.002s
I20250831 13:57:59.491469 20995 consensus_queue.cc:1035] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.492097 20691 tablet_bootstrap.cc:492] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.498066 20693 tablet_bootstrap.cc:654] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.500532 20692 tablet_bootstrap.cc:654] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.504369 20691 tablet_bootstrap.cc:654] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.512333 20995 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.554224 20954 consensus_queue.cc:1035] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250831 13:57:59.565301 20997 log.cc:927] Time spent T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20250831 13:57:59.571297 20877 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:57:59.575181 20697 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.594650 20935 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.595234 20935 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.600216 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:59.601588 20467 leader_election.cc:343] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.607246 20935 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.607955 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.609957 20691 tablet_bootstrap.cc:492] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.610249 20692 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:59.610401 20691 ts_tablet_manager.cc:1397] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.119s user 0.017s sys 0.026s
I20250831 13:57:59.610663 20692 ts_tablet_manager.cc:1397] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.130s user 0.020s sys 0.016s
I20250831 13:57:59.616532 20692 raft_consensus.cc:357] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.617286 20692 raft_consensus.cc:383] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.617694 20692 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.620708 20692 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.621897 20717 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.622085 20954 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.622370 20717 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.622491 20954 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.624281 20717 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.634508 20691 raft_consensus.cc:357] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.635228 20691 raft_consensus.cc:383] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250831 13:57:59.617799 20466 leader_election.cc:343] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.635479 20691 raft_consensus.cc:738] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.632710 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:59.638913 20691 consensus_queue.cc:260] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.639428 20692 ts_tablet_manager.cc:1428] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20250831 13:57:59.641649 20692 tablet_bootstrap.cc:492] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:59.636934 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.645617 20954 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.640329 20466 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:59.642365 20501 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:59.649773 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.656402 20691 ts_tablet_manager.cc:1428] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20250831 13:57:59.656740 20617 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:59.657275 20691 tablet_bootstrap.cc:492] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.657802 20933 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:59.658244 20933 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:57:59.658859 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 47977.7us
W20250831 13:57:59.660674 20541 leader_election.cc:343] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:59.660063 20616 leader_election.cc:343] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.662998 20692 tablet_bootstrap.cc:654] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.658614 20933 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.665750 20693 tablet_bootstrap.cc:492] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.666160 20693 ts_tablet_manager.cc:1397] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.176s user 0.029s sys 0.040s
I20250831 13:57:59.651036 20935 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.666798 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:57:59.668931 20543 leader_election.cc:343] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.670720 20543 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:59.671765 20995 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.674577 20935 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.683651 20691 tablet_bootstrap.cc:654] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.696622 20717 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.698328 20693 raft_consensus.cc:357] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.699254 20693 raft_consensus.cc:383] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.697150 20717 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.699604 20693 raft_consensus.cc:738] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.700219 20693 consensus_queue.cc:260] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.701198 20717 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.702358 20693 ts_tablet_manager.cc:1428] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.036s user 0.004s sys 0.000s
I20250831 13:57:59.703616 20693 tablet_bootstrap.cc:492] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.706768 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.706974 20935 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.708043 20935 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.709801 20935 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.697417 20933 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.713012 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:59.713608 20502 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:59.714298 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:59.714545 20617 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:59.714946 20501 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.717272 20692 tablet_bootstrap.cc:492] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:59.717712 20692 ts_tablet_manager.cc:1397] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.008s
I20250831 13:57:59.720151 20877 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250831 13:57:59.721143 20616 leader_election.cc:343] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.721189 20877 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:59.722136 20877 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.720733 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.725375 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.732769 20691 tablet_bootstrap.cc:492] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.733204 20691 ts_tablet_manager.cc:1397] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.076s user 0.009s sys 0.001s
I20250831 13:57:59.734303 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:59.734862 20939 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 13:57:59.737677 20467 leader_election.cc:343] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:59.739512 20616 leader_election.cc:343] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.739635 20691 raft_consensus.cc:357] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.740295 20691 raft_consensus.cc:383] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.740607 20691 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.741696 20501 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:59.741952 20691 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.728478 20692 raft_consensus.cc:357] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.742825 20692 raft_consensus.cc:383] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.743108 20692 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.744117 20691 ts_tablet_manager.cc:1428] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.011s user 0.002s sys 0.005s
I20250831 13:57:59.743781 20692 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.744985 20691 tablet_bootstrap.cc:492] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.747529 20617 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:59.748139 20692 ts_tablet_manager.cc:1428] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20250831 13:57:59.748281 20697 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:59.748723 20697 raft_consensus.cc:695] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:59.748968 20692 tablet_bootstrap.cc:492] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:59.750545 20697 consensus_queue.cc:237] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.749796 20933 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.766285 20693 tablet_bootstrap.cc:654] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
W20250831 13:57:59.771592 20448 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:57:59.771592 20691 tablet_bootstrap.cc:654] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.773675 20877 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.774942 20650 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:57:59.777683 20697 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.778146 20697 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.781539 20466 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:59.782578 20818 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.783972 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:59.789853 20570 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.798136 20693 tablet_bootstrap.cc:492] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.798560 20693 ts_tablet_manager.cc:1397] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.095s user 0.013s sys 0.003s
I20250831 13:57:59.793141 20818 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.796233 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:59.783349 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:57:59.800911 20693 raft_consensus.cc:357] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.801553 20693 raft_consensus.cc:383] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.801803 20693 raft_consensus.cc:738] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.802364 20693 consensus_queue.cc:260] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:59.803072 20616 leader_election.cc:343] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:59.803856 20617 leader_election.cc:343] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.804013 20693 ts_tablet_manager.cc:1428] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250831 13:57:59.804847 20693 tablet_bootstrap.cc:492] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.806538 20939 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.806222 20397 catalog_manager.cc:5582] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:57:59.810326 20693 tablet_bootstrap.cc:654] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.810660 20939 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.811210 20692 tablet_bootstrap.cc:654] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.811138 20939 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.820426 20818 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.820875 20818 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.821091 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.823009 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.823554 20650 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:57:59.822275 20877 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.819623 20939 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.829336 20466 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:57:59.829484 20818 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.830199 20818 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:59.830600 20818 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:59.831012 20818 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.795918 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.835765 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:57:59.794456 20697 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.838009 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:57:59.839195 20467 leader_election.cc:343] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:59.840368 20616 leader_election.cc:343] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.844942 20693 tablet_bootstrap.cc:492] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:57:59.845273 20693 ts_tablet_manager.cc:1397] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.006s
I20250831 13:57:59.840898 20616 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:57:59.848842 20692 tablet_bootstrap.cc:492] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:57:59.849294 20692 ts_tablet_manager.cc:1397] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.100s user 0.016s sys 0.020s
I20250831 13:57:59.849900 20691 tablet_bootstrap.cc:492] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:57:59.850267 20691 ts_tablet_manager.cc:1397] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.009s
W20250831 13:57:59.843161 20466 leader_election.cc:343] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:57:59.856048 20467 leader_election.cc:343] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.860749 20818 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.861778 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:57:59.862816 20818 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:57:59.863360 20818 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.880534 20692 raft_consensus.cc:357] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.882187 20692 raft_consensus.cc:383] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.882474 20692 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.854936 20717 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:57:59.883666 20692 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.886449 20697 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:57:59.886921 20697 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.887599 20502 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:57:59.888165 20575 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.889947 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:57:59.890520 20577 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:59.891517 20939 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.891587 20616 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:57:59.892496 20880 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:57:59.892828 20880 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:57:59.893127 20880 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.893836 20691 raft_consensus.cc:357] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.894547 20691 raft_consensus.cc:383] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.894858 20691 raft_consensus.cc:738] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.895596 20691 consensus_queue.cc:260] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.896025 20692 ts_tablet_manager.cc:1428] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.046s user 0.006s sys 0.002s
I20250831 13:57:59.896154 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:57:59.897419 20692 tablet_bootstrap.cc:492] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:57:59.897572 20650 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.898130 20691 ts_tablet_manager.cc:1428] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.048s user 0.004s sys 0.002s
I20250831 13:57:59.899670 20697 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.900137 20691 tablet_bootstrap.cc:492] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:57:59.907235 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:59.907332 20693 raft_consensus.cc:357] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.908162 20693 raft_consensus.cc:383] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:57:59.908453 20693 raft_consensus.cc:738] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:57:59.909126 20693 consensus_queue.cc:260] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.911607 20693 ts_tablet_manager.cc:1428] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.066s user 0.017s sys 0.003s
I20250831 13:57:59.913617 20693 tablet_bootstrap.cc:492] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:57:59.914471 20697 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:57:59.914427 20880 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:57:59.916831 20467 leader_election.cc:343] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:57:59.918845 20650 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:57:59.919358 20880 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:57:59.920610 20466 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:59.921708 20818 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:59.922323 20818 raft_consensus.cc:695] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:57:59.923345 20818 consensus_queue.cc:237] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.955634 20617 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:57:59.959964 20697 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:57:59.960422 20697 raft_consensus.cc:695] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:57:59.961205 20697 consensus_queue.cc:237] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:57:59.977499 20691 tablet_bootstrap.cc:654] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.977363 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:57:59.977967 20577 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:57:59.990767 20692 tablet_bootstrap.cc:654] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:57:59.993669 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:57:59.994599 20501 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:57:59.999908 20396 catalog_manager.cc:5582] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:00.000072 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:00.003364 20499 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.001016 20400 catalog_manager.cc:5582] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:00.009486 20697 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:00.009681 20499 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:00.012678 20693 tablet_bootstrap.cc:654] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.016742 20617 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.017469 21018 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:00.017866 21018 raft_consensus.cc:695] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:00.021798 20577 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:00.018608 21018 consensus_queue.cc:237] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.052942 20692 tablet_bootstrap.cc:492] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:00.053153 20691 tablet_bootstrap.cc:492] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:00.053362 20692 ts_tablet_manager.cc:1397] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.156s user 0.013s sys 0.007s
I20250831 13:58:00.053526 20691 ts_tablet_manager.cc:1397] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.154s user 0.011s sys 0.008s
I20250831 13:58:00.055864 20692 raft_consensus.cc:357] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.056540 20692 raft_consensus.cc:383] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.056813 20692 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.057224 20693 tablet_bootstrap.cc:492] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:00.057646 20693 ts_tablet_manager.cc:1397] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.144s user 0.011s sys 0.006s
I20250831 13:58:00.057484 20692 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.060158 20693 raft_consensus.cc:357] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.060822 20693 raft_consensus.cc:383] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.061100 20693 raft_consensus.cc:738] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.061760 20692 ts_tablet_manager.cc:1428] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250831 13:58:00.062623 20692 tablet_bootstrap.cc:492] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:00.062310 20691 raft_consensus.cc:357] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.062987 20691 raft_consensus.cc:383] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.063252 20691 raft_consensus.cc:738] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.064086 20691 consensus_queue.cc:260] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.064646 20400 catalog_manager.cc:5582] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:00.066444 20691 ts_tablet_manager.cc:1428] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250831 13:58:00.067365 20691 tablet_bootstrap.cc:492] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:00.071028 20693 consensus_queue.cc:260] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.073176 20693 ts_tablet_manager.cc:1428] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20250831 13:58:00.073966 20693 tablet_bootstrap.cc:492] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:00.079234 20693 tablet_bootstrap.cc:654] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.083730 20691 tablet_bootstrap.cc:654] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.088937 20693 tablet_bootstrap.cc:492] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:00.089409 20693 ts_tablet_manager.cc:1397] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.003s
I20250831 13:58:00.091703 20693 raft_consensus.cc:357] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.092280 20693 raft_consensus.cc:383] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.092494 20693 raft_consensus.cc:738] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.093099 20693 consensus_queue.cc:260] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.094349 20693 ts_tablet_manager.cc:1428] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250831 13:58:00.095207 20693 tablet_bootstrap.cc:492] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:00.102859 20691 tablet_bootstrap.cc:492] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:00.103310 20691 ts_tablet_manager.cc:1397] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20250831 13:58:00.105348 20691 raft_consensus.cc:357] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.105787 20691 raft_consensus.cc:383] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.105993 20691 raft_consensus.cc:738] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.106575 20691 consensus_queue.cc:260] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.108373 20693 tablet_bootstrap.cc:654] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.108701 20691 ts_tablet_manager.cc:1428] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250831 13:58:00.109577 20691 tablet_bootstrap.cc:492] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:00.110286 20692 tablet_bootstrap.cc:654] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.116166 20691 tablet_bootstrap.cc:654] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.124198 20691 tablet_bootstrap.cc:492] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:00.124611 20691 ts_tablet_manager.cc:1397] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20250831 13:58:00.125301 20692 tablet_bootstrap.cc:492] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:00.125705 20692 ts_tablet_manager.cc:1397] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.004s
I20250831 13:58:00.126006 20693 tablet_bootstrap.cc:492] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:00.126368 20693 ts_tablet_manager.cc:1397] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.012s
I20250831 13:58:00.126987 20691 raft_consensus.cc:357] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.127583 20691 raft_consensus.cc:383] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.127813 20691 raft_consensus.cc:738] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.127876 20692 raft_consensus.cc:357] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.128460 20692 raft_consensus.cc:383] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.128790 20692 raft_consensus.cc:738] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.128391 20691 consensus_queue.cc:260] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.128914 20693 raft_consensus.cc:357] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.129541 20693 raft_consensus.cc:383] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.129431 20692 consensus_queue.cc:260] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.129860 20693 raft_consensus.cc:738] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.130831 20691 ts_tablet_manager.cc:1428] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250831 13:58:00.130599 20693 consensus_queue.cc:260] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.131657 20691 tablet_bootstrap.cc:492] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:00.132781 20693 ts_tablet_manager.cc:1428] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:58:00.134409 20693 tablet_bootstrap.cc:492] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:00.135673 20692 ts_tablet_manager.cc:1428] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250831 13:58:00.136508 20692 tablet_bootstrap.cc:492] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:00.137929 20691 tablet_bootstrap.cc:654] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.152880 20693 tablet_bootstrap.cc:654] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.160921 20691 tablet_bootstrap.cc:492] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:00.161355 20691 ts_tablet_manager.cc:1397] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.008s
I20250831 13:58:00.165195 20691 raft_consensus.cc:357] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.166193 20691 raft_consensus.cc:383] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.166519 20691 raft_consensus.cc:738] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.168062 20692 tablet_bootstrap.cc:654] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.167403 20691 consensus_queue.cc:260] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.169958 20691 ts_tablet_manager.cc:1428] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250831 13:58:00.171054 20691 tablet_bootstrap.cc:492] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:00.171101 20693 tablet_bootstrap.cc:492] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:00.171700 20693 ts_tablet_manager.cc:1397] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.037s user 0.019s sys 0.003s
I20250831 13:58:00.174897 20693 raft_consensus.cc:357] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.175747 20693 raft_consensus.cc:383] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.176097 20693 raft_consensus.cc:738] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.176564 20691 tablet_bootstrap.cc:654] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.176961 20693 consensus_queue.cc:260] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.187280 21018 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:00.188190 20692 tablet_bootstrap.cc:492] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:00.188673 20692 ts_tablet_manager.cc:1397] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.007s
I20250831 13:58:00.192031 20693 ts_tablet_manager.cc:1428] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20250831 13:58:00.192031 20692 raft_consensus.cc:357] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.192970 20692 raft_consensus.cc:383] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.193336 20692 raft_consensus.cc:738] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.193042 20693 tablet_bootstrap.cc:492] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:00.200186 20577 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.201741 20693 tablet_bootstrap.cc:654] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.204842 20954 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.205276 20954 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.194067 20692 consensus_queue.cc:260] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.207631 20692 ts_tablet_manager.cc:1428] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250831 13:58:00.207729 20693 tablet_bootstrap.cc:492] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:00.208138 20693 ts_tablet_manager.cc:1397] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20250831 13:58:00.208590 20692 tablet_bootstrap.cc:492] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:00.208832 20717 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.209252 20717 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.210925 20954 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:00.211890 20717 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.212700 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:00.212028 20693 raft_consensus.cc:357] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.213541 20693 raft_consensus.cc:383] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.213835 20693 raft_consensus.cc:738] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.214376 20693 consensus_queue.cc:260] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.220943 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:00.221640 20693 ts_tablet_manager.cc:1428] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250831 13:58:00.222515 20693 tablet_bootstrap.cc:492] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:00.224567 20877 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.225772 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:00.225813 20877 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.228528 20877 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.229482 20692 tablet_bootstrap.cc:654] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.221647 20648 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:00.237090 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:00.238855 20954 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.241881 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:00.242504 20501 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:00.245249 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:00.247291 20693 tablet_bootstrap.cc:654] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
W20250831 13:58:00.248123 20616 leader_election.cc:343] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:58:00.249890 20616 leader_election.cc:343] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:00.250198 20717 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.250924 20697 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:00.253139 20617 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:00.254153 20697 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:00.254457 20697 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:00.254743 20697 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.262128 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:00.267701 20877 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.250938 20717 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.268200 20877 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.269869 20691 tablet_bootstrap.cc:492] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:00.270030 20717 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.262878 20501 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:00.270342 20691 ts_tablet_manager.cc:1397] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.099s user 0.017s sys 0.023s
I20250831 13:58:00.271214 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:00.265471 20697 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.272027 20499 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:00.275393 20818 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.275748 20691 raft_consensus.cc:357] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.275871 20818 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.276427 20691 raft_consensus.cc:383] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.276738 20691 raft_consensus.cc:738] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.277321 20691 consensus_queue.cc:260] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.277683 20818 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:00.274993 20939 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.278213 20939 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.279444 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:00.279804 20617 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:00.279969 20939 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:00.280184 20572 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:00.280900 20617 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.281563 20880 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:00.281883 20880 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:00.281948 20933 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:00.282186 20880 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.284694 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:00.285240 20650 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:00.286568 20467 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.287335 20939 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:00.287665 20939 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:00.290218 20939 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.289376 20466 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.282467 20933 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:00.291452 20818 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:00.291757 20933 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.291805 20818 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:00.292290 20818 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.293102 20697 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.293259 20691 ts_tablet_manager.cc:1428] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.023s user 0.008s sys 0.000s
I20250831 13:58:00.294934 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:00.295773 20577 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:00.297526 20691 tablet_bootstrap.cc:492] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:00.304939 20691 tablet_bootstrap.cc:654] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.308028 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:00.313195 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:00.314255 20501 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.313136 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:00.323169 20499 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:00.312897 20880 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.324476 20572 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:00.313891 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:00.316372 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:00.313549 20877 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.328135 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:00.332587 20692 tablet_bootstrap.cc:492] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
W20250831 13:58:00.332955 20616 leader_election.cc:343] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:00.333968 20818 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.335076 20648 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:00.336206 20692 ts_tablet_manager.cc:1397] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.128s user 0.034s sys 0.028s
I20250831 13:58:00.337072 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:00.328629 20954 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 13:58:00.340018 20616 leader_election.cc:343] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:00.340845 20617 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.340992 20933 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:00.342489 20616 leader_election.cc:343] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:00.343868 20818 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:00.344516 20933 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.345043 20933 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:00.345355 20933 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:00.345692 20933 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.346848 20877 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.347504 20877 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.349756 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:00.350044 20877 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.350327 20569 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.335217 20939 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.352917 20939 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:00.352852 20692 raft_consensus.cc:357] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.353591 20692 raft_consensus.cc:383] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.335445 20501 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:00.353904 20692 raft_consensus.cc:738] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.328569 20880 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.354537 20692 consensus_queue.cc:260] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.356124 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:00.356666 20650 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.357859 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:00.358397 20572 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.359057 20933 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.360443 20933 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.364490 20572 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:00.365383 20467 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.367766 20939 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:00.368852 20617 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:00.369705 20939 raft_consensus.cc:695] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:00.370818 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:00.370882 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:00.371150 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:00.371577 20499 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.371691 20500 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
W20250831 13:58:00.372424 20617 leader_election.cc:343] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:00.368405 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:00.373135 20501 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.373771 20692 ts_tablet_manager.cc:1428] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.037s user 0.005s sys 0.001s
I20250831 13:58:00.374621 20692 tablet_bootstrap.cc:492] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:00.374897 20877 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.375681 20877 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.377650 20877 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.375180 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:00.378870 20933 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:00.379320 20648 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.380626 20933 raft_consensus.cc:695] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:00.381094 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:00.381754 20502 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:00.382345 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:00.382930 20569 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:00.383246 20577 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:00.383044 20617 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.375213 20939 consensus_queue.cc:237] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.389252 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:00.385166 20692 tablet_bootstrap.cc:654] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.385433 20650 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:00.386595 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:00.387252 20693 tablet_bootstrap.cc:492] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:00.389393 21018 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:00.384878 20500 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:00.391590 20693 ts_tablet_manager.cc:1397] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.169s user 0.034s sys 0.004s
I20250831 13:58:00.381450 20933 consensus_queue.cc:237] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.394284 21018 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:00.400590 20467 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.401110 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:00.417831 21021 consensus_queue.cc:1035] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:00.406077 20648 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:00.407516 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:00.418944 20572 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.419108 20693 raft_consensus.cc:357] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.419750 20693 raft_consensus.cc:383] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.419996 20693 raft_consensus.cc:738] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.420620 20501 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:00.420542 20693 consensus_queue.cc:260] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.422029 20693 ts_tablet_manager.cc:1428] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.030s user 0.005s sys 0.005s
I20250831 13:58:00.408777 20880 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:00.409965 20499 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:00.416054 21018 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.417233 20818 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:00.423101 20693 tablet_bootstrap.cc:492] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:00.429756 20617 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:00.433225 20697 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
W20250831 13:58:00.433264 20616 leader_election.cc:343] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:00.433735 20697 raft_consensus.cc:695] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:00.435501 20617 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:00.434662 20697 consensus_queue.cc:237] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.426985 20880 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:00.436928 20818 raft_consensus.cc:695] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:00.440683 20691 tablet_bootstrap.cc:492] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:00.441759 20691 ts_tablet_manager.cc:1397] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.144s user 0.008s sys 0.020s
W20250831 13:58:00.444446 20616 leader_election.cc:343] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:00.442011 20880 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.451442 20717 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:00.463246 20570 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.478919 20818 consensus_queue.cc:237] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.473256 20717 raft_consensus.cc:695] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:00.499401 20692 tablet_bootstrap.cc:492] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:00.499357 20572 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:00.500020 20692 ts_tablet_manager.cc:1397] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.126s user 0.027s sys 0.030s
I20250831 13:58:00.500837 20616 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:00.501647 20697 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:00.502127 20697 raft_consensus.cc:695] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:00.497344 20691 raft_consensus.cc:357] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.502666 20691 raft_consensus.cc:383] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.504187 20691 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.502981 20697 consensus_queue.cc:237] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.504288 21018 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.504688 20692 raft_consensus.cc:357] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.504815 20691 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.505336 20692 raft_consensus.cc:383] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:00.505618 20692 raft_consensus.cc:738] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:00.506274 20692 consensus_queue.cc:260] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.507128 20691 ts_tablet_manager.cc:1428] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.065s user 0.012s sys 0.003s
I20250831 13:58:00.508061 20691 tablet_bootstrap.cc:492] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: Bootstrap starting.
W20250831 13:58:00.508404 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 72798us
I20250831 13:58:00.506093 21018 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.506983 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:00.521807 20717 consensus_queue.cc:237] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:00.526558 20616 leader_election.cc:343] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:00.528776 21046 consensus_queue.cc:1035] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:00.534032 20400 catalog_manager.cc:5582] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:00.536718 21021 consensus_queue.cc:1035] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:58:00.551337 20692 ts_tablet_manager.cc:1428] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20250831 13:58:00.552330 20692 tablet_bootstrap.cc:492] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:00.561564 21021 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.581298 20880 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.586606 20693 tablet_bootstrap.cc:654] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
W20250831 13:58:00.583353 21049 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.075s user 0.003s sys 0.000s
I20250831 13:58:00.679634 20691 tablet_bootstrap.cc:654] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.683054 20880 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.579442 21021 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.728112 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:00.728771 20499 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:00.729183 20692 tablet_bootstrap.cc:654] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:00.855755 20697 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250831 13:58:00.752475 20400 catalog_manager.cc:5582] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:00.866251 20880 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:00.866879 20880 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:00.869244 20880 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:00.873036 21021 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:00.873263 21053 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
W20250831 13:58:00.769191 21047 log.cc:927] Time spent T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.210s user 0.000s sys 0.000s
I20250831 13:58:00.795673 20397 catalog_manager.cc:5582] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:00.880941 20499 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:00.925643 20693 tablet_bootstrap.cc:492] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:00.926276 20693 ts_tablet_manager.cc:1397] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.503s user 0.042s sys 0.011s
I20250831 13:58:01.003242 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:00.979504 20877 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20250831 13:58:00.989951 20995 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:01.034960 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 57074.1us
I20250831 13:58:01.036099 20995 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.072533 20693 raft_consensus.cc:357] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.073449 20693 raft_consensus.cc:383] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:01.073777 20693 raft_consensus.cc:738] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:01.074486 20693 consensus_queue.cc:260] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:01.088414 21055 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20250831 13:58:01.092211 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:01.093047 20502 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:01.055552 20717 consensus_queue.cc:1035] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20250831 13:58:01.119853 20995 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:01.123811 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:01.124701 20576 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:01.128508 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:01.134837 20541 leader_election.cc:343] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:01.135416 20617 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:01.136610 20693 ts_tablet_manager.cc:1428] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.210s user 0.005s sys 0.002s
I20250831 13:58:01.137671 20617 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:01.138150 21058 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:01.138626 21058 raft_consensus.cc:695] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:01.140609 21018 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:01.140764 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:01.141171 21018 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:01.141361 20499 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:01.141674 21018 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:01.143457 20693 tablet_bootstrap.cc:492] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:01.139349 21058 consensus_queue.cc:237] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.156296 20499 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
W20250831 13:58:00.850778 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 148324us
I20250831 13:58:01.172780 20691 tablet_bootstrap.cc:492] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:01.173225 20691 ts_tablet_manager.cc:1397] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.665s user 0.014s sys 0.012s
I20250831 13:58:01.165184 21067 consensus_queue.cc:1035] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:58:01.174170 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:01.176128 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:01.177758 20691 raft_consensus.cc:357] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.178419 20691 raft_consensus.cc:383] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:01.178699 20691 raft_consensus.cc:738] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:01.179754 20691 consensus_queue.cc:260] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.166797 21044 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:58:01.185257 21018 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.187304 21018 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:01.188256 20576 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
W20250831 13:58:01.189682 20466 leader_election.cc:343] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:01.191957 20692 tablet_bootstrap.cc:492] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:01.192384 20692 ts_tablet_manager.cc:1397] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.640s user 0.019s sys 0.018s
I20250831 13:58:01.194828 20692 raft_consensus.cc:357] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.195443 20692 raft_consensus.cc:383] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:01.195690 20692 raft_consensus.cc:738] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:01.196262 20692 consensus_queue.cc:260] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.198638 20692 ts_tablet_manager.cc:1428] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250831 13:58:01.199497 20692 tablet_bootstrap.cc:492] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:01.204576 20939 consensus_queue.cc:1035] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:58:01.182528 20691 ts_tablet_manager.cc:1428] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250831 13:58:01.213012 20691 tablet_bootstrap.cc:492] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602: Bootstrap starting.
W20250831 13:58:01.218294 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 154555us
W20250831 13:58:01.222332 20616 consensus_peers.cc:489] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 -> Peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Couldn't send request to peer 212b66976f874aa682cae0fb77928786. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250831 13:58:01.226130 20616 leader_election.cc:343] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:01.229233 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:01.244993 21067 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
W20250831 13:58:01.271121 20616 consensus_peers.cc:489] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 -> Peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Couldn't send request to peer 212b66976f874aa682cae0fb77928786. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:01.265406 21067 raft_consensus.cc:695] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:01.284111 20692 tablet_bootstrap.cc:654] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:01.262925 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:01.303058 20691 tablet_bootstrap.cc:654] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:01.303956 21071 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.075s
I20250831 13:58:01.266630 21052 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
W20250831 13:58:01.321425 20467 leader_election.cc:343] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:01.321817 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:01.322266 20880 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:01.322460 21052 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:01.283164 21067 consensus_queue.cc:237] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.327494 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:01.329114 20693 tablet_bootstrap.cc:654] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:01.334944 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:01.335580 20495 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:01.340711 20818 consensus_queue.cc:1035] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
W20250831 13:58:01.348563 20616 consensus_peers.cc:489] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 -> Peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Couldn't send request to peer 212b66976f874aa682cae0fb77928786. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250831 13:58:01.375502 20616 leader_election.cc:343] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:01.377898 20691 tablet_bootstrap.cc:492] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:01.378526 20691 ts_tablet_manager.cc:1397] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.166s user 0.015s sys 0.011s
I20250831 13:58:01.344945 20817 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
I20250831 13:58:01.391980 21070 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.046s
I20250831 13:58:01.418625 20691 raft_consensus.cc:357] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.419711 20691 raft_consensus.cc:383] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:01.420053 20691 raft_consensus.cc:738] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:01.420855 20691 consensus_queue.cc:260] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.442586 20691 ts_tablet_manager.cc:1428] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.058s user 0.008s sys 0.000s
W20250831 13:58:01.449631 21084 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20250831 13:58:01.449699 20543 leader_election.cc:343] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:01.450266 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:01.460880 20691 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:01.505573 21057 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250831 13:58:01.595820 20448 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:58:01.640573 20692 tablet_bootstrap.cc:492] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:01.641116 20692 ts_tablet_manager.cc:1397] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.442s user 0.031s sys 0.044s
I20250831 13:58:01.653208 20933 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:01.665239 20933 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.656870 20495 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:01.676414 20933 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:01.683109 20692 raft_consensus.cc:357] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.683866 20692 raft_consensus.cc:383] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:01.684165 20692 raft_consensus.cc:738] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:01.684988 20692 consensus_queue.cc:260] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.693940 20692 ts_tablet_manager.cc:1428] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20250831 13:58:01.696888 20692 tablet_bootstrap.cc:492] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:01.697893 20995 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:01.701081 20995 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.703008 20995 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:01.705024 21091 log.cc:927] Time spent T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.182s user 0.005s sys 0.001s
I20250831 13:58:01.700117 20617 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:01.722080 20397 catalog_manager.cc:5582] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:01.727239 20397 catalog_manager.cc:5582] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:01.729547 20397 catalog_manager.cc:5582] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:01.744405 21070 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:01.744616 21065 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:01.745124 21065 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.748988 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:01.749228 21070 raft_consensus.cc:695] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:01.749806 20495 raft_consensus.cc:2391] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:01.750139 21070 consensus_queue.cc:237] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.743155 20691 tablet_bootstrap.cc:654] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:01.763671 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:01.776474 21088 log.cc:927] Time spent T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.234s user 0.004s sys 0.005s
I20250831 13:58:01.790732 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:01.791002 20616 leader_election.cc:343] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:01.793624 20693 tablet_bootstrap.cc:492] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:01.793962 20693 ts_tablet_manager.cc:1397] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.651s user 0.021s sys 0.018s
I20250831 13:58:01.799407 20693 raft_consensus.cc:357] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.800058 20693 raft_consensus.cc:383] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:01.800304 20573 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:01.800305 20693 raft_consensus.cc:738] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
W20250831 13:58:01.803015 21093 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.159s user 0.000s sys 0.002s
I20250831 13:58:01.803121 20693 consensus_queue.cc:260] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.805420 20693 ts_tablet_manager.cc:1428] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250831 13:58:01.806120 20693 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:01.788507 21065 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:01.827402 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:01.831431 21021 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:01.833248 20575 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:01.833334 21021 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:01.840174 20541 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:01.878556 20692 tablet_bootstrap.cc:654] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:01.882723 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:01.883380 20501 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:01.900032 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:01.904400 20616 leader_election.cc:343] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
W20250831 13:58:01.904842 21101 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20250831 13:58:01.975231 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 47321.7us
I20250831 13:58:01.922550 21021 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:01.975497 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 48569.8us
I20250831 13:58:01.984656 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:02.004741 20693 tablet_bootstrap.cc:654] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.006268 21068 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.006577 20691 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:02.007072 20691 ts_tablet_manager.cc:1397] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.546s user 0.031s sys 0.039s
W20250831 13:58:02.024763 20617 leader_election.cc:343] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.025235 20617 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:02.028647 20717 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.030871 21058 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.032335 21058 consensus_queue.cc:1035] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250831 13:58:02.035414 20466 leader_election.cc:343] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.044378 20691 raft_consensus.cc:357] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.045116 20691 raft_consensus.cc:383] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.045372 20691 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
W20250831 13:58:02.050192 21108 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.166s user 0.000s sys 0.003s
I20250831 13:58:02.051568 20691 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.053268 20691 ts_tablet_manager.cc:1428] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.046s user 0.004s sys 0.000s
I20250831 13:58:02.054136 20691 tablet_bootstrap.cc:492] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:02.061282 20693 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:02.061407 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.062098 20693 ts_tablet_manager.cc:1397] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.256s user 0.009s sys 0.009s
I20250831 13:58:02.062026 20570 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:02.063242 20617 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:02.065770 20693 raft_consensus.cc:357] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.066473 20693 raft_consensus.cc:383] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.066720 20693 raft_consensus.cc:738] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.067039 21046 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:02.067458 21046 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:02.067272 20693 consensus_queue.cc:260] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.068060 21046 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.069046 20693 ts_tablet_manager.cc:1428] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250831 13:58:02.069846 20693 tablet_bootstrap.cc:492] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:02.070345 20880 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.074959 21046 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.087709 20692 tablet_bootstrap.cc:492] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.076454 20880 consensus_queue.cc:1035] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.079162 20691 tablet_bootstrap.cc:654] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.080377 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:02.089340 20817 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:02.089742 20817 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:02.090070 20817 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.088169 20692 ts_tablet_manager.cc:1397] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.392s user 0.016s sys 0.003s
I20250831 13:58:02.101069 20692 raft_consensus.cc:357] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.101660 20692 raft_consensus.cc:383] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.101934 20692 raft_consensus.cc:738] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.083380 20693 tablet_bootstrap.cc:654] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.102535 20692 consensus_queue.cc:260] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.110606 20692 ts_tablet_manager.cc:1428] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250831 13:58:02.112197 20692 tablet_bootstrap.cc:492] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:02.112846 20691 tablet_bootstrap.cc:492] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:02.113297 20691 ts_tablet_manager.cc:1397] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.000s
I20250831 13:58:02.115829 20691 raft_consensus.cc:357] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.116557 20691 raft_consensus.cc:383] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.116849 20691 raft_consensus.cc:738] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.117516 20691 consensus_queue.cc:260] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.119088 20691 ts_tablet_manager.cc:1428] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250831 13:58:02.131693 20691 tablet_bootstrap.cc:492] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:02.130690 20692 tablet_bootstrap.cc:654] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.132834 21059 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.134622 20397 catalog_manager.cc:5582] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:02.140460 20397 catalog_manager.cc:5582] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:02.133684 20817 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.143543 20691 tablet_bootstrap.cc:654] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.133167 21046 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:02.155447 20817 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.143852 21059 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.164367 21059 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:02.164656 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:02.166872 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:02.167425 20497 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.167608 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:02.169258 20617 leader_election.cc:343] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.171028 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:02.171556 20570 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.177237 20692 tablet_bootstrap.cc:492] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.177783 20692 ts_tablet_manager.cc:1397] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.066s user 0.013s sys 0.000s
I20250831 13:58:02.193603 20570 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:02.196172 20400 catalog_manager.cc:5582] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:02.199431 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:02.200124 20497 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
W20250831 13:58:02.186832 20466 leader_election.cc:343] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.203361 20617 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:02.204679 21068 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.205559 20692 raft_consensus.cc:357] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.205111 21068 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.206310 20692 raft_consensus.cc:383] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.206702 20692 raft_consensus.cc:738] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.207120 20692 consensus_queue.cc:260] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.205579 21070 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:02.209179 21068 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.209703 20692 ts_tablet_manager.cc:1428] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20250831 13:58:02.207713 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:02.211583 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.211335 21070 raft_consensus.cc:695] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:02.214144 20571 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.214028 21070 consensus_queue.cc:237] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.216547 20693 tablet_bootstrap.cc:492] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:02.218127 20693 ts_tablet_manager.cc:1397] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.148s user 0.020s sys 0.027s
I20250831 13:58:02.220549 20693 raft_consensus.cc:357] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.221237 20693 raft_consensus.cc:383] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.221525 20693 raft_consensus.cc:738] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.221953 20817 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:02.222471 20817 raft_consensus.cc:695] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:02.222134 20693 consensus_queue.cc:260] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.223327 20817 consensus_queue.cc:237] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.225327 20693 ts_tablet_manager.cc:1428] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250831 13:58:02.226377 20693 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:02.210604 20692 tablet_bootstrap.cc:492] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:02.232874 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:02.234545 20543 leader_election.cc:343] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.235993 20691 tablet_bootstrap.cc:492] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:02.237704 20691 ts_tablet_manager.cc:1397] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.106s user 0.009s sys 0.009s
I20250831 13:58:02.240046 20691 raft_consensus.cc:357] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.241173 20691 raft_consensus.cc:383] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.241580 20691 raft_consensus.cc:738] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.240818 20693 tablet_bootstrap.cc:654] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.251325 20691 consensus_queue.cc:260] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.257759 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:02.260516 20691 ts_tablet_manager.cc:1428] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.023s user 0.007s sys 0.002s
W20250831 13:58:02.261044 20616 leader_election.cc:343] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.261531 20691 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:02.263809 20397 catalog_manager.cc:5582] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:02.267647 20691 tablet_bootstrap.cc:654] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
W20250831 13:58:02.268368 20616 leader_election.cc:343] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.274947 20616 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:02.275698 21044 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.276235 20880 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.276691 20880 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.279691 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:02.278550 20880 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:02.282888 20541 leader_election.cc:343] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44d163094add49049cc749547d891602 (127.18.233.131:45067): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.283303 20541 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:02.285027 20693 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:02.285095 20954 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.291042 20693 ts_tablet_manager.cc:1397] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.010s
I20250831 13:58:02.295694 20693 raft_consensus.cc:357] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.296388 20693 raft_consensus.cc:383] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.296679 20693 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.300588 20693 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.288282 20692 tablet_bootstrap.cc:654] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.304913 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.307937 20576 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:02.308677 20693 ts_tablet_manager.cc:1428] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.017s user 0.001s sys 0.003s
I20250831 13:58:02.309954 20693 tablet_bootstrap.cc:492] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:02.310899 20616 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:02.311558 21059 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.311825 20691 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:02.312474 20691 ts_tablet_manager.cc:1397] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.051s user 0.015s sys 0.015s
I20250831 13:58:02.309317 20400 catalog_manager.cc:5582] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:02.315665 20691 raft_consensus.cc:357] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.316344 20691 raft_consensus.cc:383] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.316612 20691 raft_consensus.cc:738] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.317267 20691 consensus_queue.cc:260] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.317905 21059 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.319507 20691 ts_tablet_manager.cc:1428] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250831 13:58:02.320426 20691 tablet_bootstrap.cc:492] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602: Bootstrap starting.
I20250831 13:58:02.321597 20693 tablet_bootstrap.cc:654] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.326196 20691 tablet_bootstrap.cc:654] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.328116 20717 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.328578 20717 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.330235 20717 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:02.331269 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:02.332854 20617 leader_election.cc:343] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.334192 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.335533 20692 tablet_bootstrap.cc:492] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.336155 20692 ts_tablet_manager.cc:1397] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.000s
W20250831 13:58:02.336205 20616 leader_election.cc:343] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.336620 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:02.337245 20717 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.340222 20692 raft_consensus.cc:357] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.340912 20692 raft_consensus.cc:383] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.341169 20692 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.341761 20692 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.349687 20692 ts_tablet_manager.cc:1428] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250831 13:58:02.350484 20692 tablet_bootstrap.cc:492] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:02.353170 21044 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.358520 20693 tablet_bootstrap.cc:492] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:02.358945 20693 ts_tablet_manager.cc:1397] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.007s
I20250831 13:58:02.361900 20693 raft_consensus.cc:357] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.364512 20692 tablet_bootstrap.cc:654] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.367349 20693 raft_consensus.cc:383] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.367645 20693 raft_consensus.cc:738] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.368183 20693 consensus_queue.cc:260] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.369704 20693 ts_tablet_manager.cc:1428] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250831 13:58:02.370464 20693 tablet_bootstrap.cc:492] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48: Bootstrap starting.
I20250831 13:58:02.374264 20691 tablet_bootstrap.cc:492] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602: No bootstrap required, opened a new log
I20250831 13:58:02.374719 20691 ts_tablet_manager.cc:1397] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602: Time spent bootstrapping tablet: real 0.054s user 0.028s sys 0.023s
I20250831 13:58:02.377324 20691 raft_consensus.cc:357] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.378111 20691 raft_consensus.cc:383] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.378419 20691 raft_consensus.cc:738] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.379189 20691 consensus_queue.cc:260] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.381893 20691 ts_tablet_manager.cc:1428] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250831 13:58:02.387696 20693 tablet_bootstrap.cc:654] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.392009 20692 tablet_bootstrap.cc:492] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.392402 20692 ts_tablet_manager.cc:1397] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.000s
I20250831 13:58:02.394508 20692 raft_consensus.cc:357] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.395120 20692 raft_consensus.cc:383] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.394948 21059 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.395452 20692 raft_consensus.cc:738] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.395638 21059 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.396087 20692 consensus_queue.cc:260] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.398413 20692 ts_tablet_manager.cc:1428] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250831 13:58:02.398991 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:02.399688 20495 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 0.
I20250831 13:58:02.400950 20617 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:02.401994 21044 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:02.402420 21044 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:02.402805 21044 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.409277 21044 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.410177 21059 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:02.399266 20692 tablet_bootstrap.cc:492] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:02.411489 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:02.411998 20495 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.413390 20693 tablet_bootstrap.cc:492] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48: No bootstrap required, opened a new log
I20250831 13:58:02.413771 20693 ts_tablet_manager.cc:1397] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48: Time spent bootstrapping tablet: real 0.043s user 0.012s sys 0.004s
I20250831 13:58:02.416127 20693 raft_consensus.cc:357] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.416671 20693 raft_consensus.cc:383] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.416919 20693 raft_consensus.cc:738] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.417542 20693 consensus_queue.cc:260] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.418036 21044 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:02.419754 20693 ts_tablet_manager.cc:1428] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250831 13:58:02.421586 20576 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.423120 20495 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:02.423193 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:02.423835 20568 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.424329 20617 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:02.430454 20692 tablet_bootstrap.cc:654] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.423120 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.431944 21044 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:02.438858 21044 raft_consensus.cc:695] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:02.440582 20568 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:02.445024 20880 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.448362 20568 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.451210 20692 tablet_bootstrap.cc:492] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.441066 21044 consensus_queue.cc:237] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.464515 20817 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.464996 20817 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.467223 21052 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.467628 21052 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.469245 21052 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.470122 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:02.470719 20643 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:02.471680 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.471904 20466 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:02.472910 21104 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:02.473242 21104 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:02.473557 21104 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
W20250831 13:58:02.473953 20467 leader_election.cc:343] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.474768 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:02.469893 20817 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.475358 20643 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:02.451632 20692 ts_tablet_manager.cc:1397] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.000s
I20250831 13:58:02.476882 20466 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:02.478068 20935 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:02.478714 20935 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:02.479135 20935 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.480756 20692 raft_consensus.cc:357] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.482656 20692 raft_consensus.cc:383] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.482990 20692 raft_consensus.cc:738] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.483675 20692 consensus_queue.cc:260] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.472754 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.480837 20396 catalog_manager.cc:5582] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:02.508000 20571 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:02.515951 20692 ts_tablet_manager.cc:1428] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20250831 13:58:02.508365 21104 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.517163 20692 tablet_bootstrap.cc:492] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:02.518363 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:02.519649 20467 leader_election.cc:343] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.517686 21104 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.509806 20935 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.524879 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:02.525403 20643 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.525337 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:02.526039 20576 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.536243 20576 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:02.537994 20643 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:02.538991 20466 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:02.540160 21052 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:02.540516 20935 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.540597 21052 raft_consensus.cc:695] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:02.541010 20467 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:02.541501 21052 consensus_queue.cc:237] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.545590 20692 tablet_bootstrap.cc:654] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.548453 21104 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:02.548939 21104 raft_consensus.cc:695] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:02.549785 21104 consensus_queue.cc:237] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.551934 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:02.552469 20643 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.564167 20692 tablet_bootstrap.cc:492] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.564626 20692 ts_tablet_manager.cc:1397] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.003s
I20250831 13:58:02.572412 20643 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:02.573215 20692 raft_consensus.cc:357] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.573844 20692 raft_consensus.cc:383] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.574070 20692 raft_consensus.cc:738] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.574836 21052 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.575297 21052 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.574627 20692 consensus_queue.cc:260] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.579044 20692 ts_tablet_manager.cc:1428] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20250831 13:58:02.579934 20692 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:02.582028 21052 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.585090 20692 tablet_bootstrap.cc:654] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.592594 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:02.592933 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.593206 20648 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
W20250831 13:58:02.598788 20467 leader_election.cc:343] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.597518 20397 catalog_manager.cc:5582] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:02.604885 20397 catalog_manager.cc:5582] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:02.611606 20466 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:02.612798 21052 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:02.613147 21052 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:02.613445 21052 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.669095 21052 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.676698 21052 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.678560 20692 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.679080 20692 ts_tablet_manager.cc:1397] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.099s user 0.038s sys 0.050s
I20250831 13:58:02.679275 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:02.681377 20467 leader_election.cc:343] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Illegal state: must be running to vote when last-logged opid is not known
I20250831 13:58:02.681524 20692 raft_consensus.cc:357] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.682233 20692 raft_consensus.cc:383] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.682135 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:02.682601 20692 raft_consensus.cc:738] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.682838 20643 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.683300 20692 consensus_queue.cc:260] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.685444 20692 ts_tablet_manager.cc:1428] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250831 13:58:02.690330 20692 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:02.693485 21057 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.693994 21057 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.695632 20692 tablet_bootstrap.cc:654] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.695878 21057 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.698304 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:02.696030 20448 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250831 13:58:02.699069 20648 raft_consensus.cc:2391] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:02.698932 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:02.701112 20541 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:02.701766 21068 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.709342 21052 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:58:02.711372 21068 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.711351 20643 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:02.713300 20466 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:02.714182 20935 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:02.714607 20935 raft_consensus.cc:695] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:02.715252 20692 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.715745 20692 ts_tablet_manager.cc:1397] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.005s
I20250831 13:58:02.715433 20935 consensus_queue.cc:237] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.716506 21104 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.718058 20692 raft_consensus.cc:357] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.718822 20692 raft_consensus.cc:383] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.719106 20692 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.719899 20692 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.722884 20692 ts_tablet_manager.cc:1428] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.007s user 0.002s sys 0.005s
I20250831 13:58:02.723937 20692 tablet_bootstrap.cc:492] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786: Bootstrap starting.
I20250831 13:58:02.731271 20692 tablet_bootstrap.cc:654] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786: Neither blocks nor log segments found. Creating new log.
I20250831 13:58:02.736675 21104 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.737303 21104 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.739790 21104 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.742801 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:02.743538 20650 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:02.744850 20466 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:02.745756 21104 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250831 13:58:02.746259 21104 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:02.747784 21057 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.750797 21057 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.752226 20573 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.748204 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:02.755676 20568 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 0.
I20250831 13:58:02.756349 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:02.761912 20501 raft_consensus.cc:2391] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:02.763154 21057 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.765535 21059 consensus_queue.cc:1035] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250831 13:58:02.746671 21104 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.786715 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:02.789215 20541 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:02.796272 21104 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.798976 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:02.799602 20568 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.818389 21066 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.829082 21046 consensus_queue.cc:1035] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.831674 20692 tablet_bootstrap.cc:492] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786: No bootstrap required, opened a new log
I20250831 13:58:02.832151 20692 ts_tablet_manager.cc:1397] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786: Time spent bootstrapping tablet: real 0.108s user 0.012s sys 0.002s
I20250831 13:58:02.838400 20692 raft_consensus.cc:357] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.839356 20692 raft_consensus.cc:383] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250831 13:58:02.839644 20692 raft_consensus.cc:738] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Initialized, Role: FOLLOWER
I20250831 13:58:02.840221 20692 consensus_queue.cc:260] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.842464 20692 ts_tablet_manager.cc:1428] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250831 13:58:02.843535 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:02.844137 20643 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.843427 20396 catalog_manager.cc:5582] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:02.836692 21104 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.851748 20568 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:02.852993 20467 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:02.853857 21066 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:02.854311 21066 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.856125 21066 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:02.857482 21104 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader election won for term 1
I20250831 13:58:02.858112 21104 raft_consensus.cc:695] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:02.859077 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:02.862514 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:02.868956 20650 raft_consensus.cc:2391] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 1: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:02.871541 20543 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:02.873000 21066 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 0 FOLLOWER]: Advancing to term 1
I20250831 13:58:02.876085 21104 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:02.892972 21066 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250831 13:58:02.893836 20643 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:02.913046 20400 catalog_manager.cc:5582] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:02.929803 21070 consensus_queue.cc:1035] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.941687 21046 consensus_queue.cc:1035] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.951869 21052 consensus_queue.cc:1035] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:02.973789 20935 consensus_queue.cc:1035] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:03.117180 20935 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:03.120771 20935 consensus_queue.cc:1035] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:03.128510 21052 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:03.135699 21052 consensus_queue.cc:1035] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:03.358388 20935 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:03.371758 21052 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250831 13:58:04.016597 19366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250831 13:58:04.040598 21164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:58:04.040763 21167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250831 13:58:04.040666 21165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250831 13:58:04.042254 19366 server_base.cc:1047] running on GCE node
I20250831 13:58:04.044663 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250831 13:58:04.045019 19366 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250831 13:58:04.045305 19366 hybrid_clock.cc:648] HybridClock initialized: now 1756648684045286 us; error 0 us; skew 500 ppm
I20250831 13:58:04.046109 19366 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250831 13:58:04.049331 19366 webserver.cc:480] Webserver started at http://127.18.233.132:35343/ using document root <none> and password file <none>
I20250831 13:58:04.049813 19366 fs_manager.cc:362] Metadata directory not provided
I20250831 13:58:04.050010 19366 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250831 13:58:04.050279 19366 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250831 13:58:04.051386 19366 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-3-root/instance:
uuid: "ab7ade51fa134a1b814fab748f01d43d"
format_stamp: "Formatted at 2025-08-31 13:58:04 on dist-test-slave-sg9n"
I20250831 13:58:04.056308 19366 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250831 13:58:04.060062 21172 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250831 13:58:04.060833 19366 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250831 13:58:04.061097 19366 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-3-root
uuid: "ab7ade51fa134a1b814fab748f01d43d"
format_stamp: "Formatted at 2025-08-31 13:58:04 on dist-test-slave-sg9n"
I20250831 13:58:04.061394 19366 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250831 13:58:04.087409 19366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250831 13:58:04.089093 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250831 13:58:04.090646 19366 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250831 13:58:04.092967 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250831 13:58:04.093214 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250831 13:58:04.093487 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20250831 13:58:04.093662 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250831 13:58:04.158855 19366 rpc_server.cc:307] RPC server started. Bound to: 127.18.233.132:35173
I20250831 13:58:04.158963 21234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.18.233.132:35173 every 8 connection(s)
I20250831 13:58:04.171013 19366 auto_leader_rebalancer-test.cc:417] add a tserver: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:04.185381 21235 heartbeater.cc:344] Connected to a master server at 127.18.233.190:42663
I20250831 13:58:04.185820 21235 heartbeater.cc:461] Registering TS with master...
I20250831 13:58:04.186494 21235 heartbeater.cc:507] Master 127.18.233.190:42663 requested a full tablet report, sending...
I20250831 13:58:04.188060 20400 ts_manager.cc:194] Registered new tserver with Master: ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:04.189198 20400 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54880
I20250831 13:58:04.845028 20501 consensus_queue.cc:237] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:04.852291 20648 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:04.853947 20817 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:04.860634 20573 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:04.862599 21242 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:04.863790 21245 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:04.863629 20648 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:04.869908 20573 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:04.885041 20396 catalog_manager.cc:5582] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:04.894570 20648 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:04.906039 20573 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:04.909052 21239 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 13:58:04.914463 20468 consensus_peers.cc:489] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 -> Peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Couldn't send request to peer ab7ade51fa134a1b814fab748f01d43d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62fa3a4f7ea043fdbcffdcc84f9c28c0. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:04.923954 21239 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:04.925872 20568 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:04.957999 20501 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:04.965759 20498 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:04.975965 21059 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:04.977917 20397 catalog_manager.cc:5582] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 reported cstate change: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:04.990748 20648 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:04.992385 20568 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 13:58:04.993261 20468 consensus_peers.cc:489] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 -> Peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Couldn't send request to peer ab7ade51fa134a1b814fab748f01d43d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 346d047c73c648a5aab2f44efa73aee7. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:04.993933 21243 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:04.994990 20935 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 13:58:05.018307 20617 consensus_peers.cc:489] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 -> Peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Couldn't send request to peer ab7ade51fa134a1b814fab748f01d43d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 512ab0e0a72d4872b147e21acc0832fd. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:05.025051 20935 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:05.025517 20498 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:05.034543 20648 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:05.049142 20573 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:05.052923 20400 catalog_manager.cc:5582] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 reported cstate change: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:05.070570 20573 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.090827 20648 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:05.094384 21240 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:05.100481 20498 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:05.101559 21240 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250831 13:58:05.121052 20543 consensus_peers.cc:489] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 -> Peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Couldn't send request to peer ab7ade51fa134a1b814fab748f01d43d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca1477952e8c40adb14c71db5a7a4b92. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:05.125299 21057 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:05.126510 20648 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:05.127077 20498 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:05.140661 20397 catalog_manager.cc:5582] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 reported cstate change: config changed from index -1 to 3, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:05.191663 21235 heartbeater.cc:499] Master 127.18.233.190:42663 was elected leader, sending a full tablet report...
I20250831 13:58:05.272372 21273 ts_tablet_manager.cc:927] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:05.275300 21273 tablet_copy_client.cc:323] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.233.129:35131
I20250831 13:58:05.295800 20512 tablet_copy_service.cc:140] P 58aded50ab0d4609a053647eb11b1f48: Received BeginTabletCopySession request for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 from peer ab7ade51fa134a1b814fab748f01d43d ({username='slave'} at 127.0.0.1:37606)
I20250831 13:58:05.296424 20512 tablet_copy_service.cc:161] P 58aded50ab0d4609a053647eb11b1f48: Beginning new tablet copy session on tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 from peer ab7ade51fa134a1b814fab748f01d43d at {username='slave'} at 127.0.0.1:37606: session id = ab7ade51fa134a1b814fab748f01d43d-62fa3a4f7ea043fdbcffdcc84f9c28c0
I20250831 13:58:05.309883 20512 tablet_copy_source_session.cc:215] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 13:58:05.313560 21273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:58:05.327772 21273 tablet_copy_client.cc:806] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 0 data blocks...
I20250831 13:58:05.328399 21273 tablet_copy_client.cc:670] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 1 WAL segments...
I20250831 13:58:05.332502 21273 tablet_copy_client.cc:538] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 13:58:05.340134 21273 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap starting.
I20250831 13:58:05.366243 21273 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 13:58:05.367146 21273 tablet_bootstrap.cc:492] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap complete.
I20250831 13:58:05.367772 21273 ts_tablet_manager.cc:1397] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.006s
I20250831 13:58:05.370671 21273 raft_consensus.cc:357] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.371496 21273 raft_consensus.cc:738] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Initialized, Role: LEARNER
I20250831 13:58:05.372109 21273 consensus_queue.cc:260] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.385504 21273 ts_tablet_manager.cc:1428] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Time spent starting tablet: real 0.017s user 0.006s sys 0.012s
I20250831 13:58:05.393496 20512 tablet_copy_service.cc:342] P 58aded50ab0d4609a053647eb11b1f48: Request end of tablet copy session ab7ade51fa134a1b814fab748f01d43d-62fa3a4f7ea043fdbcffdcc84f9c28c0 received from {username='slave'} at 127.0.0.1:37606
I20250831 13:58:05.394016 20512 tablet_copy_service.cc:434] P 58aded50ab0d4609a053647eb11b1f48: ending tablet copy session ab7ade51fa134a1b814fab748f01d43d-62fa3a4f7ea043fdbcffdcc84f9c28c0 on tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 with peer ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:05.394179 21278 ts_tablet_manager.cc:927] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:05.396652 21278 tablet_copy_client.cc:323] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.233.129:35131
I20250831 13:58:05.397943 20512 tablet_copy_service.cc:140] P 58aded50ab0d4609a053647eb11b1f48: Received BeginTabletCopySession request for tablet 346d047c73c648a5aab2f44efa73aee7 from peer ab7ade51fa134a1b814fab748f01d43d ({username='slave'} at 127.0.0.1:37606)
I20250831 13:58:05.398396 20512 tablet_copy_service.cc:161] P 58aded50ab0d4609a053647eb11b1f48: Beginning new tablet copy session on tablet 346d047c73c648a5aab2f44efa73aee7 from peer ab7ade51fa134a1b814fab748f01d43d at {username='slave'} at 127.0.0.1:37606: session id = ab7ade51fa134a1b814fab748f01d43d-346d047c73c648a5aab2f44efa73aee7
I20250831 13:58:05.405512 20512 tablet_copy_source_session.cc:215] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 13:58:05.408404 21278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346d047c73c648a5aab2f44efa73aee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:58:05.421976 21278 tablet_copy_client.cc:806] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 0 data blocks...
I20250831 13:58:05.422533 21278 tablet_copy_client.cc:670] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 1 WAL segments...
I20250831 13:58:05.426452 21278 tablet_copy_client.cc:538] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 13:58:05.433386 21278 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap starting.
I20250831 13:58:05.455199 21278 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 13:58:05.455978 21278 tablet_bootstrap.cc:492] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap complete.
I20250831 13:58:05.456521 21278 ts_tablet_manager.cc:1397] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.007s
I20250831 13:58:05.459363 21278 raft_consensus.cc:357] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.460300 21278 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Initialized, Role: LEARNER
I20250831 13:58:05.460978 21278 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.463325 21278 ts_tablet_manager.cc:1428] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250831 13:58:05.465279 21273 ts_tablet_manager.cc:927] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:05.465610 20512 tablet_copy_service.cc:342] P 58aded50ab0d4609a053647eb11b1f48: Request end of tablet copy session ab7ade51fa134a1b814fab748f01d43d-346d047c73c648a5aab2f44efa73aee7 received from {username='slave'} at 127.0.0.1:37606
I20250831 13:58:05.466018 20512 tablet_copy_service.cc:434] P 58aded50ab0d4609a053647eb11b1f48: ending tablet copy session ab7ade51fa134a1b814fab748f01d43d-346d047c73c648a5aab2f44efa73aee7 on tablet 346d047c73c648a5aab2f44efa73aee7 with peer ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:05.470219 21273 tablet_copy_client.cc:323] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.233.131:45067
I20250831 13:58:05.484467 20661 tablet_copy_service.cc:140] P 44d163094add49049cc749547d891602: Received BeginTabletCopySession request for tablet 512ab0e0a72d4872b147e21acc0832fd from peer ab7ade51fa134a1b814fab748f01d43d ({username='slave'} at 127.0.0.1:46464)
I20250831 13:58:05.485030 20661 tablet_copy_service.cc:161] P 44d163094add49049cc749547d891602: Beginning new tablet copy session on tablet 512ab0e0a72d4872b147e21acc0832fd from peer ab7ade51fa134a1b814fab748f01d43d at {username='slave'} at 127.0.0.1:46464: session id = ab7ade51fa134a1b814fab748f01d43d-512ab0e0a72d4872b147e21acc0832fd
I20250831 13:58:05.491221 20661 tablet_copy_source_session.cc:215] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 13:58:05.494801 21273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 512ab0e0a72d4872b147e21acc0832fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:58:05.507326 21273 tablet_copy_client.cc:806] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 0 data blocks...
I20250831 13:58:05.507826 21273 tablet_copy_client.cc:670] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 1 WAL segments...
I20250831 13:58:05.510881 21273 tablet_copy_client.cc:538] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 13:58:05.517517 21273 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Bootstrap starting.
I20250831 13:58:05.534121 21273 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 13:58:05.534806 21273 tablet_bootstrap.cc:492] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Bootstrap complete.
I20250831 13:58:05.535341 21273 ts_tablet_manager.cc:1397] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.003s
I20250831 13:58:05.538543 21273 raft_consensus.cc:357] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.539271 21273 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Initialized, Role: LEARNER
I20250831 13:58:05.539810 21273 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.541584 21273 ts_tablet_manager.cc:1428] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250831 13:58:05.542968 20661 tablet_copy_service.cc:342] P 44d163094add49049cc749547d891602: Request end of tablet copy session ab7ade51fa134a1b814fab748f01d43d-512ab0e0a72d4872b147e21acc0832fd received from {username='slave'} at 127.0.0.1:46464
I20250831 13:58:05.543255 20661 tablet_copy_service.cc:434] P 44d163094add49049cc749547d891602: ending tablet copy session ab7ade51fa134a1b814fab748f01d43d-512ab0e0a72d4872b147e21acc0832fd on tablet 512ab0e0a72d4872b147e21acc0832fd with peer ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:05.681427 21210 raft_consensus.cc:1215] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 13:58:05.711119 21273 ts_tablet_manager.cc:927] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:05.712721 21273 tablet_copy_client.cc:323] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.233.130:44629
I20250831 13:58:05.723428 20587 tablet_copy_service.cc:140] P 212b66976f874aa682cae0fb77928786: Received BeginTabletCopySession request for tablet ca1477952e8c40adb14c71db5a7a4b92 from peer ab7ade51fa134a1b814fab748f01d43d ({username='slave'} at 127.0.0.1:34436)
I20250831 13:58:05.723938 20587 tablet_copy_service.cc:161] P 212b66976f874aa682cae0fb77928786: Beginning new tablet copy session on tablet ca1477952e8c40adb14c71db5a7a4b92 from peer ab7ade51fa134a1b814fab748f01d43d at {username='slave'} at 127.0.0.1:34436: session id = ab7ade51fa134a1b814fab748f01d43d-ca1477952e8c40adb14c71db5a7a4b92
I20250831 13:58:05.730326 20587 tablet_copy_source_session.cc:215] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 13:58:05.732733 21273 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1477952e8c40adb14c71db5a7a4b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:58:05.745114 21273 tablet_copy_client.cc:806] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 0 data blocks...
I20250831 13:58:05.745582 21273 tablet_copy_client.cc:670] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 1 WAL segments...
I20250831 13:58:05.748788 21273 tablet_copy_client.cc:538] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 13:58:05.753818 21273 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap starting.
I20250831 13:58:05.771266 21273 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 13:58:05.772033 21273 tablet_bootstrap.cc:492] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap complete.
I20250831 13:58:05.772557 21273 ts_tablet_manager.cc:1397] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.001s
I20250831 13:58:05.774541 21273 raft_consensus.cc:357] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.775259 21273 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Initialized, Role: LEARNER
I20250831 13:58:05.775770 21273 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:05.777573 21273 ts_tablet_manager.cc:1428] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250831 13:58:05.778986 20587 tablet_copy_service.cc:342] P 212b66976f874aa682cae0fb77928786: Request end of tablet copy session ab7ade51fa134a1b814fab748f01d43d-ca1477952e8c40adb14c71db5a7a4b92 received from {username='slave'} at 127.0.0.1:34436
I20250831 13:58:05.779314 20587 tablet_copy_service.cc:434] P 212b66976f874aa682cae0fb77928786: ending tablet copy session ab7ade51fa134a1b814fab748f01d43d-ca1477952e8c40adb14c71db5a7a4b92 on tablet ca1477952e8c40adb14c71db5a7a4b92 with peer ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:05.936714 21210 raft_consensus.cc:1215] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 13:58:05.959725 21210 raft_consensus.cc:1215] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 13:58:06.214294 21242 raft_consensus.cc:1062] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:06.216734 21242 consensus_queue.cc:237] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:06.222517 20648 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.223774 21210 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.224252 20568 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.224249 21243 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:06.225924 20935 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:06.226670 21243 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:06.271811 21210 raft_consensus.cc:1215] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250831 13:58:06.277693 20648 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.279234 21243 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.291585 20573 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.299351 21210 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.298863 20400 catalog_manager.cc:5582] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 reported cstate change: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.331389 20495 consensus_queue.cc:237] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:06.336582 20573 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 13:58:06.338361 21242 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:06.340778 21210 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 13:58:06.342006 21242 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:06.345705 21243 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 44d163094add49049cc749547d891602 (127.18.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.347038 20573 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 44d163094add49049cc749547d891602 (127.18.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.356151 20387 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 13:58:06.358958 21210 raft_consensus.cc:2953] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 44d163094add49049cc749547d891602 (127.18.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.363698 20398 catalog_manager.cc:5582] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: config changed from index 3 to 4, VOTER 44d163094add49049cc749547d891602 (127.18.233.131) evicted. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:58:06.373172 21264 raft_consensus.cc:1062] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:06.380915 21264 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:06.395902 20636 tablet_service.cc:1515] Processing DeleteTablet for tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 with delete_type TABLET_DATA_TOMBSTONED (TS 44d163094add49049cc749547d891602 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57684
I20250831 13:58:06.406059 21209 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.408046 21059 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:06.410728 20570 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.412525 21059 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:06.420979 20498 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.421787 21301 tablet_replica.cc:331] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: stopping tablet replica
I20250831 13:58:06.422580 21264 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:06.437268 21301 raft_consensus.cc:2241] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 13:58:06.438076 21301 raft_consensus.cc:2270] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 13:58:06.444329 21301 ts_tablet_manager.cc:1905] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 13:58:06.450058 21287 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.459102 21301 ts_tablet_manager.cc:1918] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 13:58:06.458097 20570 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.459581 21301 log.cc:1199] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 44d163094add49049cc749547d891602: Deleting WAL directory at /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-2-root/wals/62fa3a4f7ea043fdbcffdcc84f9c28c0
I20250831 13:58:06.461530 20385 catalog_manager.cc:4928] TS 44d163094add49049cc749547d891602 (127.18.233.131:45067): tablet 62fa3a4f7ea043fdbcffdcc84f9c28c0 (table test-workload [id=bf9639b2cb72453d8b67d48646dc2938]) successfully deleted
I20250831 13:58:06.465737 20495 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.468201 21242 raft_consensus.cc:1062] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:06.472637 20398 catalog_manager.cc:5582] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 reported cstate change: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:58:06.470595 21242 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:06.455899 21209 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.492751 20648 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.493372 20570 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.493597 21209 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:06.494375 21290 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:06.495195 20935 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:06.499686 20935 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:06.504014 20648 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:06.514405 21210 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 13:58:06.515993 21059 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:06.518954 20935 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.519876 20570 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 13:58:06.521776 21059 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:06.523279 20570 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.527347 21287 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.527468 21210 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.529376 21209 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.531916 20571 raft_consensus.cc:2953] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.538743 20385 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 512ab0e0a72d4872b147e21acc0832fd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 13:58:06.549221 20400 catalog_manager.cc:5582] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 reported cstate change: config changed from index 3 to 4, VOTER 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) evicted. New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:58:06.558262 20648 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.558888 20398 catalog_manager.cc:5582] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.565025 20486 tablet_service.cc:1515] Processing DeleteTablet for tablet 512ab0e0a72d4872b147e21acc0832fd with delete_type TABLET_DATA_TOMBSTONED (TS 58aded50ab0d4609a053647eb11b1f48 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52966
I20250831 13:58:06.582132 21309 tablet_replica.cc:331] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: stopping tablet replica
I20250831 13:58:06.581810 20498 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:06.583191 21309 raft_consensus.cc:2241] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 13:58:06.583863 21309 raft_consensus.cc:2270] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 13:58:06.592649 21209 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 13:58:06.592672 20648 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 13:58:06.592851 21309 ts_tablet_manager.cc:1905] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 13:58:06.594785 20935 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20250831 13:58:06.597736 21290 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:06.608536 21242 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 212b66976f874aa682cae0fb77928786 (127.18.233.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.611843 21309 ts_tablet_manager.cc:1918] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 13:58:06.610963 21209 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 212b66976f874aa682cae0fb77928786 (127.18.233.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.612308 21309 log.cc:1199] T 512ab0e0a72d4872b147e21acc0832fd P 58aded50ab0d4609a053647eb11b1f48: Deleting WAL directory at /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-0-root/wals/512ab0e0a72d4872b147e21acc0832fd
I20250831 13:58:06.613866 20387 catalog_manager.cc:4928] TS 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): tablet 512ab0e0a72d4872b147e21acc0832fd (table test-workload [id=bf9639b2cb72453d8b67d48646dc2938]) successfully deleted
I20250831 13:58:06.614192 20644 raft_consensus.cc:2953] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 212b66976f874aa682cae0fb77928786 (127.18.233.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.618065 20387 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 346d047c73c648a5aab2f44efa73aee7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 13:58:06.626369 20397 catalog_manager.cc:5582] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: config changed from index 3 to 4, VOTER 212b66976f874aa682cae0fb77928786 (127.18.233.130) evicted. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.638279 20558 tablet_service.cc:1515] Processing DeleteTablet for tablet 346d047c73c648a5aab2f44efa73aee7 with delete_type TABLET_DATA_TOMBSTONED (TS 212b66976f874aa682cae0fb77928786 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41392
I20250831 13:58:06.650898 21310 tablet_replica.cc:331] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: stopping tablet replica
I20250831 13:58:06.652014 21310 raft_consensus.cc:2241] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250831 13:58:06.652693 21310 raft_consensus.cc:2270] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250831 13:58:06.656361 21310 ts_tablet_manager.cc:1905] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 13:58:06.669519 21310 ts_tablet_manager.cc:1918] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 13:58:06.669878 21310 log.cc:1199] T 346d047c73c648a5aab2f44efa73aee7 P 212b66976f874aa682cae0fb77928786: Deleting WAL directory at /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-1-root/wals/346d047c73c648a5aab2f44efa73aee7
I20250831 13:58:06.671914 20386 catalog_manager.cc:4928] TS 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): tablet 346d047c73c648a5aab2f44efa73aee7 (table test-workload [id=bf9639b2cb72453d8b67d48646dc2938]) successfully deleted
I20250831 13:58:06.891629 21294 raft_consensus.cc:1062] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:06.893683 21294 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:06.899718 21209 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 LEARNER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250831 13:58:06.901422 21240 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:06.901787 20644 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250831 13:58:06.903623 20498 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250831 13:58:06.904892 21294 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:06.906818 21312 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:06.930181 21312 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.931962 21209 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.935250 20648 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.942143 20498 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.949287 20396 catalog_manager.cc:5582] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: config changed from index 3 to 4, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.976146 20570 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:06.980659 21209 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:06.982544 21312 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:06.988631 20648 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:06.993233 21312 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:06.992797 21209 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:06.992152 21240 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:07.006726 20648 raft_consensus.cc:2953] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:07.015837 20386 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ca1477952e8c40adb14c71db5a7a4b92 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 13:58:07.025151 20397 catalog_manager.cc:5582] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: config changed from index 4 to 5, VOTER 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) evicted. New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:07.034544 20486 tablet_service.cc:1515] Processing DeleteTablet for tablet ca1477952e8c40adb14c71db5a7a4b92 with delete_type TABLET_DATA_TOMBSTONED (TS 58aded50ab0d4609a053647eb11b1f48 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52966
I20250831 13:58:07.035468 21309 tablet_replica.cc:331] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: stopping tablet replica
I20250831 13:58:07.036291 21309 raft_consensus.cc:2241] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:58:07.036819 21309 raft_consensus.cc:2270] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:58:07.039815 21309 ts_tablet_manager.cc:1905] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 13:58:07.050412 21309 ts_tablet_manager.cc:1918] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20250831 13:58:07.050740 21309 log.cc:1199] T ca1477952e8c40adb14c71db5a7a4b92 P 58aded50ab0d4609a053647eb11b1f48: Deleting WAL directory at /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-0-root/wals/ca1477952e8c40adb14c71db5a7a4b92
I20250831 13:58:07.051806 20387 catalog_manager.cc:4928] TS 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): tablet ca1477952e8c40adb14c71db5a7a4b92 (table test-workload [id=bf9639b2cb72453d8b67d48646dc2938]) successfully deleted
I20250831 13:58:08.092093 20498 consensus_queue.cc:237] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:08.097915 20573 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:08.098011 20644 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 13:58:08.098866 20468 consensus_peers.cc:489] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 -> Peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Couldn't send request to peer ab7ade51fa134a1b814fab748f01d43d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00c91580fd1b4c45a280862eedb8e1ac. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:08.099329 20935 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:08.100106 21242 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:08.106432 20935 raft_consensus.cc:2953] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.107880 20573 raft_consensus.cc:2953] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.109275 20644 raft_consensus.cc:2953] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.117003 20498 consensus_queue.cc:237] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:08.124936 20400 catalog_manager.cc:5582] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:08.138141 20648 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250831 13:58:08.140197 20468 consensus_peers.cc:489] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 -> Peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Couldn't send request to peer ab7ade51fa134a1b814fab748f01d43d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90a0a56228d04660b3cb351485be6593. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:08.141534 20935 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:08.145378 20573 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:08.147385 21317 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:08.195523 21242 raft_consensus.cc:2953] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.197832 20570 raft_consensus.cc:2953] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.200026 20644 raft_consensus.cc:2953] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.209679 20570 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:08.214185 20396 catalog_manager.cc:5582] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 reported cstate change: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
W20250831 13:58:08.220577 20543 consensus_peers.cc:489] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 -> Peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Couldn't send request to peer ab7ade51fa134a1b814fab748f01d43d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c14599239fd4c99b3331692a513a092. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:08.220851 20644 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:08.221210 20498 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:08.222394 21318 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:08.223229 21327 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:08.239190 21318 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.242539 20498 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.256090 20648 consensus_queue.cc:237] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:08.257391 20644 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.258364 20397 catalog_manager.cc:5582] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
W20250831 13:58:08.273917 20617 consensus_peers.cc:489] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 -> Peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Couldn't send request to peer ab7ade51fa134a1b814fab748f01d43d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c47fb589f98457ab81d63e6b9d19fd0. This is attempt 1: this message will repeat every 5th retry.
I20250831 13:58:08.278509 20497 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:08.279815 21330 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:08.280236 20570 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250831 13:58:08.281877 21287 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:08.289819 21330 raft_consensus.cc:2953] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.291421 20570 raft_consensus.cc:2953] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.301442 20497 raft_consensus.cc:2953] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.308712 20400 catalog_manager.cc:5582] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 reported cstate change: config changed from index -1 to 2, NON_VOTER ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) added. New cstate: current_term: 1 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:08.912880 21344 ts_tablet_manager.cc:927] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:09.258437 21344 tablet_copy_client.cc:323] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.233.131:45067
I20250831 13:58:09.610731 21346 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:09.611532 21346 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:09.614558 21346 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:09.616513 21346 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:09.617231 21346 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:09.619546 21346 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:09.985564 21346 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:09.986421 21346 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:09.987052 21348 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:09.988801 21346 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.009707 21346 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.011677 21348 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.011039 21346 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.013732 21349 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.015010 21346 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.012879 21350 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.015116 21349 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.015899 21350 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.016486 21346 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.016992 21355 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.018277 21349 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.018654 21348 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.018790 21349 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.014264 21352 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.022228 21352 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.024766 21352 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.025395 21352 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.026079 21352 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.028290 21352 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.021497 21350 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.015552 21354 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.030247 21354 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.032831 21354 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.034061 21354 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.017027 21346 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.035580 21352 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.036123 21351 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.038151 21351 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.034581 21354 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:10.017477 21355 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.019192 21348 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.042363 21348 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.046131 21348 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.046904 21348 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.047569 21348 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.050071 21348 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.050736 21348 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.037150 21346 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.033492 21350 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.039552 21356 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.033087 21353 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.051431 21348 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.052279 21353 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:10.054550 21348 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.051728 21350 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.065563 21350 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.043602 21355 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.043977 21354 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:10.028926 21349 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.072211 21355 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.072769 21355 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.073921 21349 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.036633 21352 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:10.074776 21355 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.075799 21352 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.053608 21356 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.102662 21353 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.042845 21351 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.119943 21356 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.225150 21356 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.226141 21356 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:10.228893 21356 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:10.253276 20523 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0a20 after lost signal to thread 19369
I20250831 13:58:10.259589 21352 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.351208 21352 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.353155 21352 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:10.348232 20543 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.477s user 0.013s sys 0.007s
W20250831 13:58:10.353926 20543 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.483s user 0.013s sys 0.007s
I20250831 13:58:10.355824 21353 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.357317 21353 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.358386 21349 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.359038 21354 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.359127 21349 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.359622 21354 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:10.360652 21352 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.361135 21352 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.361495 21354 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:10.361981 21354 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.362385 21354 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.363166 21318 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.363678 21318 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.364248 21354 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.364655 21338 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.365199 21338 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.364763 21354 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.365864 21349 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.366292 21354 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.368530 21352 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.369206 21338 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.365867 21318 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.370491 21359 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.370638 21318 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.369714 21354 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.358314 21351 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.371063 21359 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.375450 21318 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.375595 21351 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:10.376884 21359 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.377450 21359 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.377609 21318 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.377985 21359 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.378396 21351 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.379544 21359 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.379436 21350 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.380249 21350 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.382560 21350 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.383066 21353 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.383603 21353 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.384119 21353 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.385089 21358 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.386390 21359 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.384398 21357 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.386749 21353 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.387182 21359 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.439877 21359 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.387980 21318 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.445041 21318 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:10.424849 20542 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.738s user 0.018s sys 0.014s
I20250831 13:58:10.385692 21358 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.387405 21360 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:10.454246 20542 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.767s user 0.019s sys 0.014s
I20250831 13:58:10.436878 21357 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.482553 20935 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.483376 20935 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.485572 20935 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.473573 21362 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.477564 21363 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.492013 21363 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:10.494642 21340 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:10.495323 20523 debug-util.cc:398] Leaking SignalData structure 0x7b08001dc1c0 after lost signal to thread 20434
I20250831 13:58:10.495201 21340 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.496433 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:10.496937 20523 debug-util.cc:398] Leaking SignalData structure 0x7b080029fd40 after lost signal to thread 20526
I20250831 13:58:10.497110 20500 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.497242 21340 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.498636 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.499312 21240 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:10.499789 20540 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.759s user 0.013s sys 0.031s
I20250831 13:58:10.499861 21240 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.511020 21240 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.512982 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.513417 21287 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.514108 21287 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.517503 21287 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.489899 21360 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.523305 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:10.511920 20540 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.823s user 0.013s sys 0.031s
I20250831 13:58:10.524199 20648 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.527712 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.532428 21287 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.533047 21287 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.535235 21353 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.536010 21353 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.538455 21353 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.543594 21353 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.545279 21353 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.547878 21353 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.550297 20935 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.550700 21240 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.557627 20935 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.557852 21240 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.559814 20935 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.560114 21240 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.560295 20935 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.560712 21240 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.561045 20935 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.561267 21240 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.563277 20935 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.564231 21363 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.530413 20500 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.566630 21240 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.567643 21364 ts_tablet_manager.cc:927] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.568087 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.571468 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.572311 21287 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.516719 21327 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.576896 21365 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.577613 21365 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:10.580590 21365 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.589322 21240 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.589599 21287 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.590102 21287 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:10.590018 21240 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.591784 21366 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.592700 21240 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.593147 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.596458 20501 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.598171 21287 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:10.600263 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.601249 20498 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.587929 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:10.593117 21338 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:10.594043 20523 debug-util.cc:398] Leaking SignalData structure 0x7b08000ee2c0 after lost signal to thread 20601
I20250831 13:58:10.604197 21287 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.595196 21355 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.592284 21366 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:10.523303 21362 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:10.607735 21369 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.606194 21287 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.609231 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.610222 21240 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.600687 21349 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.611303 20650 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 1.
I20250831 13:58:10.602329 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.584643 21327 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.610625 21340 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.610530 21356 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.611328 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.601971 20661 tablet_copy_service.cc:140] P 44d163094add49049cc749547d891602: Received BeginTabletCopySession request for tablet 2c47fb589f98457ab81d63e6b9d19fd0 from peer ab7ade51fa134a1b814fab748f01d43d ({username='slave'} at 127.0.0.1:46464)
I20250831 13:58:10.614147 21240 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.607964 21355 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:10.621079 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.608448 21359 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:10.636101 20523 debug-util.cc:398] Leaking SignalData structure 0x7b080005b5e0 after lost signal to thread 20675
I20250831 13:58:10.636446 21287 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.636366 21240 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.636929 21287 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:10.637188 20523 debug-util.cc:398] Leaking SignalData structure 0x7b0800304380 after lost signal to thread 21234
I20250831 13:58:10.637457 21287 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.608999 21338 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.642452 21287 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.644165 21240 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.642189 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.644646 21240 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:10.626922 21352 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.646118 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.629705 21318 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.627710 21209 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:10.619774 20497 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.635051 20661 tablet_copy_service.cc:161] P 44d163094add49049cc749547d891602: Beginning new tablet copy session on tablet 2c47fb589f98457ab81d63e6b9d19fd0 from peer ab7ade51fa134a1b814fab748f01d43d at {username='slave'} at 127.0.0.1:46464: session id = ab7ade51fa134a1b814fab748f01d43d-2c47fb589f98457ab81d63e6b9d19fd0
I20250831 13:58:10.649362 21240 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.652364 21338 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.652212 21371 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.652935 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:10.615911 21349 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.619485 21356 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.657064 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.658034 20501 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.638139 21359 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.646922 21346 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.626238 21340 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:10.650988 21353 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.659839 21364 tablet_copy_client.cc:323] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.233.130:44629
I20250831 13:58:10.653451 21371 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.619621 21369 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.648730 21352 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.674126 21362 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.663478 21357 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.637478 21375 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.660277 21358 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.681027 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.685370 20617 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:10.685943 21356 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.677693 21376 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.680559 21366 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.670219 21174 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:10.680466 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.682910 21240 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:10.661510 21360 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.694175 21240 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:10.688211 21318 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.666718 21346 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.693696 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.696450 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.667583 21353 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.691308 21373 ts_tablet_manager.cc:927] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
W20250831 13:58:10.701313 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 63197.8us
I20250831 13:58:10.691092 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.700948 21327 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.711777 21346 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.700376 21375 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.704283 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.712427 21352 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.712544 20643 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 3.
I20250831 13:58:10.705041 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.691080 21368 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.695101 21355 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.707361 21359 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.715975 21375 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.715795 21174 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:10.713974 21352 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:10.719060 21240 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:10.717151 21371 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.730146 21342 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.730818 21362 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:10.714901 21366 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:10.718791 21327 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.702275 21318 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.739969 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.733520 21362 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:10.751936 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 78997.2us
I20250831 13:58:10.752977 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.753332 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.753885 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.754053 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.736186 21366 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.792191 21369 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.700814 21376 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.794353 21372 ts_tablet_manager.cc:927] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:10.795367 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.800062 21376 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.751206 20661 tablet_copy_source_session.cc:215] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 13:58:10.797516 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.805197 21353 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.805852 21240 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.754092 21362 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:10.817651 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.807724 21352 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:10.808248 21359 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.810066 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.810662 21371 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:10.811081 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.811776 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.813503 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.793440 21340 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.817718 21240 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:10.755245 21349 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:10.757007 21366 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:10.754206 21327 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.736735 21368 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.753582 21342 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.868443 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.878504 20648 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.857357 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.845997 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.841890 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:10.881305 20568 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 1.
I20250831 13:58:10.818416 21352 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:10.824726 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:10.884490 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 199218us
I20250831 13:58:10.824152 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:10.834251 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:10.882957 21359 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:10.897233 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.899953 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.900696 20650 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.847086 21318 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.846189 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.835280 20587 tablet_copy_service.cc:140] P 212b66976f874aa682cae0fb77928786: Received BeginTabletCopySession request for tablet 0c14599239fd4c99b3331692a513a092 from peer ab7ade51fa134a1b814fab748f01d43d ({username='slave'} at 127.0.0.1:34436)
I20250831 13:58:10.819914 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.845257 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.844825 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.844925 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.905625 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.970255 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.970649 20648 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.942595 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.972749 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.915390 21240 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:10.971414 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.975772 20650 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.949077 20541 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:10.979583 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 122593us
I20250831 13:58:10.910203 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.953379 21344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c47fb589f98457ab81d63e6b9d19fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:58:10.987668 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.989784 20650 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.995971 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.987392 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.912766 21342 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:10.971846 20646 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:10.903376 20572 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 1.
I20250831 13:58:10.936466 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.999331 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.906801 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.000337 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.970939 21373 tablet_copy_client.cc:323] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.233.129:35131
I20250831 13:58:10.999747 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.999491 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:10.833993 21378 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.008219 20650 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:11.012490 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.987280 21318 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.013023 20541 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:10.895236 21371 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.024231 20541 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:10.987147 20587 tablet_copy_service.cc:161] P 212b66976f874aa682cae0fb77928786: Beginning new tablet copy session on tablet 0c14599239fd4c99b3331692a513a092 from peer ab7ade51fa134a1b814fab748f01d43d at {username='slave'} at 127.0.0.1:34436: session id = ab7ade51fa134a1b814fab748f01d43d-0c14599239fd4c99b3331692a513a092
I20250831 13:58:11.016255 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:10.999372 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.018086 21359 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.014532 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.023856 21342 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.024168 21318 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.029002 20541 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:11.034049 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.007797 21368 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.037195 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.036866 20541 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:11.030798 21382 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.039710 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.042562 20541 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:11.018649 21366 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:11.045928 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.037341 21318 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.035899 21342 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.036087 21327 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.039031 21359 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.041651 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.032824 21378 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.050154 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.111227 21240 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.290891 21327 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.274061 21352 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:11.291404 21327 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.200040 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.290632 21240 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.294621 20643 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
W20250831 13:58:11.087136 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 391140us
I20250831 13:58:11.295147 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.303702 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.304013 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.304487 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.305552 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.306422 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.309288 20541 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:11.309288 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.301733 21240 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.118409 21387 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.048375 21382 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.312762 21387 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.312983 20512 tablet_copy_service.cc:140] P 58aded50ab0d4609a053647eb11b1f48: Received BeginTabletCopySession request for tablet 00c91580fd1b4c45a280862eedb8e1ac from peer ab7ade51fa134a1b814fab748f01d43d ({username='slave'} at 127.0.0.1:37606)
I20250831 13:58:11.313153 21387 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.313558 20512 tablet_copy_service.cc:161] P 58aded50ab0d4609a053647eb11b1f48: Beginning new tablet copy session on tablet 00c91580fd1b4c45a280862eedb8e1ac from peer ab7ade51fa134a1b814fab748f01d43d at {username='slave'} at 127.0.0.1:37606: session id = ab7ade51fa134a1b814fab748f01d43d-00c91580fd1b4c45a280862eedb8e1ac
I20250831 13:58:11.316368 21382 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.107468 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.050365 21368 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.318210 21368 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.318291 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.318596 21368 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.319011 20575 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:11.319079 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.046368 21379 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.319708 21379 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.320072 21379 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.299201 21327 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.060984 21342 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.061013 21376 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.320786 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.316901 21382 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.046250 21340 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.202442 20587 tablet_copy_source_session.cc:215] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 13:58:11.297489 21359 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.269663 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.103006 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.125697 21369 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.088117 21372 tablet_copy_client.cc:323] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.233.129:35131
I20250831 13:58:11.122290 21371 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.129513 21360 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.102159 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.321426 21376 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.325786 21382 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.331707 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.348510 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.162032 21362 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:11.332702 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.314200 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:11.351745 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 56086.8us
I20250831 13:58:11.328320 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.328076 21340 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.352718 21369 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:11.363134 21366 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.364174 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.364406 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.381654 20570 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:11.382391 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.382869 20570 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:11.365311 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.386977 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.381520 20576 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:11.365657 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.369042 21376 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.387792 20512 tablet_copy_source_session.cc:215] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 13:58:11.366467 21352 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:11.378787 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 53633.8us
I20250831 13:58:11.379844 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.379984 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.389097 20571 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:11.380508 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.376580 20543 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.384145 21369 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.384749 21340 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.385744 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.387068 21362 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.370971 21360 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.391407 21340 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.391803 21340 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.392158 21340 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.392480 21360 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.328218 21359 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.347702 21378 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:11.395303 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 86066us
I20250831 13:58:11.404496 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.405078 20541 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:11.375450 21382 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.398694 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.399899 20543 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.401475 21340 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.403194 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.411183 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.410310 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:11.411976 20572 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:11.412657 21352 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.416033 21342 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.418066 21342 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.418754 20573 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.423100 21368 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.424404 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:11.425040 21368 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.425189 21209 raft_consensus.cc:2391] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
W20250831 13:58:11.428448 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 126401us
I20250831 13:58:11.428627 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.429704 21378 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.430100 21378 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.430446 21378 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.430711 21387 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.435299 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.435746 20573 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.434526 21388 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.436609 21387 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.436563 20512 tablet_copy_service.cc:140] P 58aded50ab0d4609a053647eb11b1f48: Received BeginTabletCopySession request for tablet 90a0a56228d04660b3cb351485be6593 from peer ab7ade51fa134a1b814fab748f01d43d ({username='slave'} at 127.0.0.1:37606)
I20250831 13:58:11.445640 20572 raft_consensus.cc:1238] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.445781 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.445979 20512 tablet_copy_service.cc:161] P 58aded50ab0d4609a053647eb11b1f48: Beginning new tablet copy session on tablet 90a0a56228d04660b3cb351485be6593 from peer ab7ade51fa134a1b814fab748f01d43d at {username='slave'} at 127.0.0.1:37606: session id = ab7ade51fa134a1b814fab748f01d43d-90a0a56228d04660b3cb351485be6593
I20250831 13:58:11.446362 20573 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:11.436571 21388 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.447106 21388 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.443266 21373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00c91580fd1b4c45a280862eedb8e1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:58:11.444741 20543 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.449999 20543 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.444131 21318 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.451151 20543 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.452414 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.453972 20543 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.454849 21360 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.455094 21318 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.455518 21369 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.456702 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.458034 20543 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.459234 20543 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.461012 21369 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.461783 21360 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.462203 21360 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.462607 21360 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.463388 20543 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.464377 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.433843 21379 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.466496 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.466751 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.467516 21369 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.467932 21369 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.468343 21369 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.470897 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.467923 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.471406 20543 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.471655 20570 raft_consensus.cc:2391] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:11.469179 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.458458 21387 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.451907 21342 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.451099 21368 consensus_queue.cc:1046] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20250831 13:58:11.479111 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.479270 20568 raft_consensus.cc:1238] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.477960 20543 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.477546 21318 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.479820 20570 raft_consensus.cc:2391] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:11.481963 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.482326 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.484697 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.483177 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.485635 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.485746 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.487051 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.487290 20574 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:11.489645 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.490473 20571 raft_consensus.cc:2391] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:11.494220 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.499809 21240 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.502867 21379 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.477803 21340 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.508711 21209 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:11.510226 21359 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:11.512531 20575 raft_consensus.cc:1238] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.513006 21390 raft_consensus.cc:3053] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:11.479415 21342 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.507313 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.507385 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.506848 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.514940 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.516368 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.517161 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.518438 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.519157 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:11.519542 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.520345 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.520555 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.521168 21360 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.521486 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.522344 21209 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.522773 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:11.524210 21384 consensus_queue.cc:1046] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:11.524803 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.525132 21384 raft_consensus.cc:3053] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:11.525449 21384 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:11.478201 21387 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.513408 21390 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:11.528123 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.531239 21360 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.533111 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.546746 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.534296 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.547431 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.547726 20512 tablet_copy_source_session.cc:215] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Tablet Copy: opened 0 blocks and 1 log segments
I20250831 13:58:11.546212 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.548715 20572 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:11.531141 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.531196 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.550014 21369 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.550709 21359 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.532745 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.551283 21372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90a0a56228d04660b3cb351485be6593. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:58:11.549685 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.552058 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.572958 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.531577 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.553196 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.555534 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.532361 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:11.535187 21378 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.576520 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.574272 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.578017 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.579170 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.580090 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.563305 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.558996 21387 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.581971 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.556470 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.582744 20568 raft_consensus.cc:2391] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:11.533098 21388 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.566469 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.531252 21384 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:11.567164 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.515952 21342 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.600378 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.567742 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.568377 21382 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.599949 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:11.615010 21384 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.571735 20543 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.615830 20576 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:11.616195 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 44557.5us
I20250831 13:58:11.570552 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.583457 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.584056 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.620365 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.525928 21344 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:47742 (request call id 15) took 2625 ms. Trace:
I20250831 13:58:11.623143 20616 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.622038 20574 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.624246 20616 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.625209 20616 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.625789 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.626588 20616 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.625154 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.625516 21368 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.627365 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.627962 21380 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.628209 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.628574 21380 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.629135 21380 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.629549 20574 raft_consensus.cc:1238] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.631323 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.631536 21369 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.635193 20574 raft_consensus.cc:1238] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.635756 21373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:47732 (request call id 31) took 1083 ms. Trace:
I20250831 13:58:11.636096 21373 rpcz_store.cc:276] 0831 13:58:10.552875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:10.556127 (+ 3252us) service_pool.cc:225] Handling call
0831 13:58:10.718885 (+162758us) ts_tablet_manager.cc:928] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
0831 13:58:11.438730 (+719845us) tablet_copy_client.cc:368] Tablet copy session begun
0831 13:58:11.536045 (+ 97315us) tablet_metadata.cc:641] Metadata flushed
0831 13:58:11.536328 (+ 283us) tablet_copy_client.cc:497] Wrote new tablet metadata
0831 13:58:11.635171 (+ 98843us) tablet_copy_client.cc:418] Wrote new consensus metadata
0831 13:58:11.635396 (+ 225us) tablet_replica.cc:206] Creating consensus instance
0831 13:58:11.635627 (+ 231us) ts_tablet_manager.cc:983] Replica registered.
0831 13:58:11.635681 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":275,"spinlock_wait_cycles":1721216,"tablet-copy.queue_time_us":62934,"thread_start_us":269574,"threads_started":2}
I20250831 13:58:11.637599 21373 tablet_copy_client.cc:806] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 0 data blocks...
I20250831 13:58:11.622748 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:11.584726 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.582944 20572 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:11.585448 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.640316 20571 raft_consensus.cc:2391] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:11.587072 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.590911 21390 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:11.595589 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.644963 21368 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.645432 21390 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:11.595978 21175 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.596248 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.596779 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.597568 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.598541 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.598654 21387 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.599649 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.609112 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.596302 21388 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.622318 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:11.639369 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:11.647512 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.649236 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.649525 21378 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.657456 21318 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.657851 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.659281 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.649921 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.652535 20568 raft_consensus.cc:2391] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:11.655131 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.655597 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.661907 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.638098 21373 tablet_copy_client.cc:670] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 1 WAL segments...
I20250831 13:58:11.664112 20569 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.664188 21387 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:11.660478 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 64595.6us
I20250831 13:58:11.654989 21362 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.670750 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.668483 21382 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.666015 20571 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.670593 21362 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.671835 21362 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.666994 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.675343 21342 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.676805 20572 raft_consensus.cc:1238] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.677546 21342 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.638808 21344 rpcz_store.cc:276] 0831 13:58:08.888108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:08.888267 (+ 159us) service_pool.cc:225] Handling call
0831 13:58:08.913261 (+ 24994us) ts_tablet_manager.cc:928] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 44d163094add49049cc749547d891602 (127.18.233.131:45067)
0831 13:58:10.929825 (+2016564us) tablet_copy_client.cc:368] Tablet copy session begun
0831 13:58:11.389577 (+459752us) tablet_metadata.cc:641] Metadata flushed
0831 13:58:11.389843 (+ 266us) tablet_copy_client.cc:497] Wrote new tablet metadata
0831 13:58:11.449898 (+ 60055us) tablet_copy_client.cc:418] Wrote new consensus metadata
0831 13:58:11.470074 (+ 20176us) tablet_replica.cc:206] Creating consensus instance
0831 13:58:11.504874 (+ 34800us) ts_tablet_manager.cc:983] Replica registered.
0831 13:58:11.508153 (+ 3279us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":1925,"tablet-copy.queue_time_us":23806,"thread_start_us":365961,"threads_started":2}
I20250831 13:58:11.678766 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:11.679080 21344 tablet_copy_client.cc:806] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 0 data blocks...
I20250831 13:58:11.679629 21344 tablet_copy_client.cc:670] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 1 WAL segments...
I20250831 13:58:11.668092 20616 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.681217 20616 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.681716 21287 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.682294 20616 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.667953 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.669734 21175 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.683948 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.683321 20616 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.684226 21366 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.670192 21380 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.685421 20616 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:11.686581 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 67330.1us
I20250831 13:58:11.687348 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.688197 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.688896 20467 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.691263 20616 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.693109 21380 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.693506 21389 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.693629 21352 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.693917 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.694341 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.694938 20616 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.685149 21355 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.694591 20643 raft_consensus.cc:2391] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:11.696159 20616 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.696883 21377 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.698364 21352 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.687142 21349 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.686609 21391 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.695652 21364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c14599239fd4c99b3331692a513a092. 1 dirs total, 0 dirs full, 0 dirs failed
I20250831 13:58:11.698818 21389 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.686046 21366 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.698213 20616 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.704560 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.704680 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.703622 21366 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.706290 21352 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.691821 21362 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:11.646952 21340 consensus_queue.cc:1162] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab7ade51fa134a1b814fab748f01d43d" responder_term: 4 status { last_received { term: 3 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 13:58:11.707939 21380 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.709995 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.711025 20616 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.712129 21392 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.713354 21377 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.713699 21392 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.714238 21392 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.714598 21287 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.715560 20616 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.716657 20616 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.717293 21352 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.718295 20616 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.719018 21352 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.720043 20616 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.720702 21352 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.721696 20616 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.722343 21352 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.716250 21287 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.723387 20616 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.724061 21352 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.724218 21362 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:11.731647 21287 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.735508 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.739190 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.739707 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:11.741601 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 52069.5us
I20250831 13:58:11.742460 20467 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.743289 21384 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.743683 20467 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.744269 21352 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.744958 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.746331 20467 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.746654 20541 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.747972 20541 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.748962 20541 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.750126 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.751353 20541 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.752521 20541 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.753278 20541 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.756028 21287 consensus_queue.cc:1046] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:11.756204 20616 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.744544 21384 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.757709 20616 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.757923 20541 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.727632 20577 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:11.754539 21387 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.761315 20541 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:11.760295 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 107004us
I20250831 13:58:11.761147 21287 raft_consensus.cc:3053] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:11.754729 21342 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.755487 21390 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.754825 21388 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.754931 21382 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.755710 21340 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.762106 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.766474 21353 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.759719 21356 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.759591 21174 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.773952 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.758783 21379 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.800876 21287 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:11.759083 21384 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.762522 21368 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.811529 21377 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.803659 20569 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.805052 20572 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.813622 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:11.815250 21384 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.806231 20616 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.776589 21387 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.816911 21375 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.817245 21375 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:11.817561 21375 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.762351 21338 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.819237 20616 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.778476 21389 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.786062 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:11.826009 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 79773.5us
I20250831 13:58:11.827600 20467 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.828547 20467 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.821720 21287 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.829511 20467 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.829766 21287 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.830495 20467 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:11.798381 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 44148.9us
I20250831 13:58:11.801537 21362 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.822832 21349 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.836700 21389 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.862756 21393 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.872835 21393 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.873219 21393 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.837965 21394 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 13:58:11.875869 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 45438.9us
I20250831 13:58:11.857468 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:11.878054 20467 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.880559 20467 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:11.842422 21174 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.862147 21395 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.879791 21394 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.896888 20467 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.884946 21391 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.905726 21384 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.906764 20501 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.908794 21352 consensus_queue.cc:1046] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:11.909729 21352 raft_consensus.cc:3053] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:11.910022 21352 raft_consensus.cc:738] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:11.910744 21352 consensus_queue.cc:260] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.911798 21352 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.900911 20467 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.917428 20467 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:11.918871 20467 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:58:11.919095 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 120019us
I20250831 13:58:11.925225 21398 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.925660 21398 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.926007 21398 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.900980 21394 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:11.926496 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 88840us
I20250831 13:58:11.913190 21375 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.937937 20616 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:11.916716 21395 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.924932 21389 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.926035 21384 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:11.916962 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 79993.5us
I20250831 13:58:11.942658 21344 tablet_copy_client.cc:538] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 13:58:11.943212 21390 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:11.943732 21390 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.944969 21372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:47732 (request call id 32) took 1391 ms. Trace:
I20250831 13:58:11.915112 21392 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:11.900671 21362 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:11.932497 21380 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.940404 21377 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.960695 21393 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:11.953104 21375 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:11.954633 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:11.945358 21372 rpcz_store.cc:276] 0831 13:58:10.553304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:10.553459 (+ 155us) service_pool.cc:225] Handling call
0831 13:58:10.731224 (+177765us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b58000c0098. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7882 00007fb9c26fab29 00007fb9c185dea8 00007fb9c191f833 00007fb9c1950ede 00007fb9c1936ddd 00007fb9c1936d89 00007fb9c1936d19 00007fb9c1936ce1
0831 13:58:10.794819 (+ 63595us) ts_tablet_manager.cc:928] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
0831 13:58:11.549905 (+755086us) tablet_copy_client.cc:368] Tablet copy session begun
0831 13:58:11.718751 (+168846us) tablet_metadata.cc:641] Metadata flushed
0831 13:58:11.719025 (+ 274us) tablet_copy_client.cc:497] Wrote new tablet metadata
0831 13:58:11.944169 (+225144us) tablet_copy_client.cc:418] Wrote new consensus metadata
0831 13:58:11.944440 (+ 271us) tablet_replica.cc:206] Creating consensus instance
0831 13:58:11.944764 (+ 324us) ts_tablet_manager.cc:983] Replica registered.
0831 13:58:11.944912 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":51673,"spinlock_wait_cycles":3479296,"tablet-copy.queue_time_us":32765,"thread_start_us":268952,"threads_started":2}
I20250831 13:58:11.956375 21390 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.966099 21372 tablet_copy_client.cc:806] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 0 data blocks...
I20250831 13:58:11.967541 21372 tablet_copy_client.cc:670] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 1 WAL segments...
I20250831 13:58:11.960345 21392 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:11.950812 21397 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 13:58:11.971333 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 51637.3us
I20250831 13:58:11.954643 21349 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.971854 20495 raft_consensus.cc:1238] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.972952 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:11.938202 21366 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:11.976572 21287 consensus_queue.cc:1046] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:11.978137 21287 raft_consensus.cc:3053] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:11.939896 21353 consensus_queue.cc:1046] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20250831 13:58:11.978523 21287 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:11.979391 21287 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:11.981564 21349 consensus_queue.cc:1046] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:11.982115 21287 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:11.983594 20616 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:11.990125 21399 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:11.977293 21338 consensus_queue.cc:1046] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20250831 13:58:11.994004 21208 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:11.998425 21402 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:11.999187 21287 consensus_queue.cc:1046] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:12.002584 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:11.995237 21375 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:11.999684 21402 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:12.043673 21402 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.038285 21393 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:12.038636 21391 raft_consensus.cc:3053] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:11.997699 21380 consensus_queue.cc:1046] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20250831 13:58:12.040292 21384 raft_consensus.cc:3053] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.049221 21384 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.049840 21344 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap starting.
I20250831 13:58:12.043020 21366 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:12.042620 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:12.047634 21391 raft_consensus.cc:738] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.048489 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.040916 21381 raft_consensus.cc:3053] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.055014 21381 raft_consensus.cc:738] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.055822 21381 consensus_queue.cc:260] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.057053 21381 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.057801 21174 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.058624 21366 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.059208 21391 consensus_queue.cc:260] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.060577 21391 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.063999 21364 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:47746 (request call id 13) took 1707 ms. Trace:
I20250831 13:58:12.064368 21364 rpcz_store.cc:276] 0831 13:58:10.356312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:10.425874 (+ 69562us) service_pool.cc:225] Handling call
0831 13:58:10.586903 (+161029us) ts_tablet_manager.cc:928] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Initiating tablet copy from peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
0831 13:58:11.685999 (+1099096us) tablet_copy_client.cc:368] Tablet copy session begun
0831 13:58:11.951985 (+265986us) tablet_metadata.cc:641] Metadata flushed
0831 13:58:11.952300 (+ 315us) tablet_copy_client.cc:497] Wrote new tablet metadata
0831 13:58:12.063312 (+111012us) tablet_copy_client.cc:418] Wrote new consensus metadata
0831 13:58:12.063605 (+ 293us) tablet_replica.cc:206] Creating consensus instance
0831 13:58:12.063885 (+ 280us) ts_tablet_manager.cc:983] Replica registered.
0831 13:58:12.063945 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":19610,"tablet-copy.queue_time_us":140891,"thread_start_us":78091,"threads_started":2}
I20250831 13:58:12.065621 21364 tablet_copy_client.cc:806] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 0 data blocks...
I20250831 13:58:12.066099 21364 tablet_copy_client.cc:670] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Starting download of 1 WAL segments...
I20250831 13:58:12.071231 21380 raft_consensus.cc:3053] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
W20250831 13:58:12.051162 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 131344us
I20250831 13:58:12.051555 21287 raft_consensus.cc:3053] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.044518 21375 raft_consensus.cc:695] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.054220 21384 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.072898 21287 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.071610 21380 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.078480 21375 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:12.089742 21377 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.091027 20645 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.091105 21380 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.082166 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.101809 21287 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.102895 21287 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.108829 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:12.097144 21384 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.114064 20616 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.114940 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:12.117380 21380 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.117928 21402 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.120977 21402 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:12.123548 20574 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.125357 21394 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.126418 20501 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.127336 21394 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:12.127622 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 45541.4us
I20250831 13:58:12.127912 21394 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.128459 21394 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:12.129405 21402 raft_consensus.cc:3053] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.129760 21402 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.130533 21402 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.131669 21402 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.134836 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:12.137336 21355 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.141352 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:12.140355 20467 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:12.138955 21403 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:12.115273 21398 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.145622 20400 catalog_manager.cc:5582] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 3 to 4, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 4 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:12.144119 21403 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:12.146660 21395 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:12.151358 21395 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:12.150967 20650 raft_consensus.cc:1238] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.149991 20643 raft_consensus.cc:1238] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.151683 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:12.146613 21364 tablet_copy_client.cc:538] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 13:58:12.151389 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:12.165287 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:12.151721 21395 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.174145 21392 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:12.174309 21377 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:58:12.174585 21392 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:12.176024 20648 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:12.175370 21377 raft_consensus.cc:3053] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.178104 21398 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:12.178952 21403 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:12.182546 20466 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:12.163972 21399 consensus_queue.cc:1046] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:12.170433 21389 consensus_queue.cc:1046] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20250831 13:58:12.172169 21397 consensus_queue.cc:1046] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20250831 13:58:12.164141 21176 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.177342 21373 tablet_copy_client.cc:538] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 13:58:12.182617 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:12.183439 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:12.190465 21389 raft_consensus.cc:3053] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.187016 21400 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:12.190886 21389 raft_consensus.cc:738] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.191139 21400 raft_consensus.cc:695] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.191649 21389 consensus_queue.cc:260] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.189752 21399 raft_consensus.cc:3053] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.191999 21400 consensus_queue.cc:237] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:12.193590 21399 raft_consensus.cc:738] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.183277 21404 consensus_queue.cc:1046] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20250831 13:58:12.195081 21399 consensus_queue.cc:260] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.195459 21397 raft_consensus.cc:3053] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.195981 21397 raft_consensus.cc:738] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.196314 21399 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.196920 21389 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.197115 21366 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.197947 21372 tablet_copy_client.cc:538] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250831 13:58:12.181605 21392 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.181598 21377 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.196933 21397 consensus_queue.cc:260] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.212891 21397 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.208839 21404 raft_consensus.cc:3053] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.214953 21404 raft_consensus.cc:738] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.216440 20648 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:12.217981 21404 consensus_queue.cc:260] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.222086 21404 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.220455 20466 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:12.204593 21377 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:12.226359 21377 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.229174 21392 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.233711 21349 consensus_queue.cc:1046] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250831 13:58:12.224097 21353 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:12.235312 21353 raft_consensus.cc:695] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.236466 21353 consensus_queue.cc:237] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.238008 21364 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap starting.
I20250831 13:58:12.234829 21349 raft_consensus.cc:3053] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.238899 21349 raft_consensus.cc:738] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.239727 21349 consensus_queue.cc:260] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:12.242292 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:12.244210 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 79009.6us
I20250831 13:58:12.247738 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.249197 20467 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.251822 20501 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.252544 20467 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.275976 21373 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Bootstrap starting.
I20250831 13:58:12.282636 20467 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.255864 21384 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.267426 21403 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.300251 21392 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:12.268635 21349 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.253480 21399 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.304636 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:12.286566 21406 log.cc:927] Time spent T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.133s user 0.000s sys 0.004s
I20250831 13:58:12.306631 20467 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.309068 20496 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.309441 21375 consensus_queue.cc:1046] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:12.293630 21395 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.286751 21208 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.313537 21402 consensus_queue.cc:1046] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20250831 13:58:12.319962 21402 raft_consensus.cc:3053] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.320389 21402 raft_consensus.cc:738] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.321319 21402 consensus_queue.cc:260] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.310510 21375 raft_consensus.cc:3053] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.340458 21375 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.342424 21395 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:12.343261 21380 consensus_queue.cc:1046] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:12.344208 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:12.315354 21400 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.323632 21372 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap starting.
I20250831 13:58:12.324380 21344 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 13:58:12.305517 21404 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.344369 21380 raft_consensus.cc:3053] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.318218 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.287213 21408 consensus_queue.cc:1046] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.100s
I20250831 13:58:12.341760 21375 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:12.360921 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44322.1us
I20250831 13:58:12.362324 20543 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.363546 20543 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.365195 20543 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.366103 20543 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.369130 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.370635 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.371016 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.371464 20571 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:12.340015 21392 consensus_queue.cc:1046] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250831 13:58:12.368522 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.358407 21380 raft_consensus.cc:738] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.370251 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:12.361835 21400 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.371905 21344 tablet_bootstrap.cc:492] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap complete.
I20250831 13:58:12.380228 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.367611 21387 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.381111 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.370218 21402 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.381974 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.372334 21390 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.384639 21390 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.381184 20577 raft_consensus.cc:2391] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:12.373253 21393 raft_consensus.cc:3053] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.373049 21379 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.375572 20496 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.391184 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:12.393182 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:12.395538 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:12.396358 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:12.397022 20498 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:12.397835 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:12.372993 20645 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:12.403102 20467 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.408510 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.410832 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:12.411762 21396 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.412065 21399 raft_consensus.cc:3053] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.412501 21399 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.413358 21399 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.414541 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:12.408519 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:12.416728 20650 raft_consensus.cc:1238] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.418176 21394 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.418586 21398 consensus_queue.cc:1046] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:12.420265 21208 raft_consensus.cc:1238] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.387929 21393 raft_consensus.cc:738] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.421520 21393 consensus_queue.cc:260] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:12.422080 20466 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:12.422901 21394 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:12.423406 21394 raft_consensus.cc:695] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.424304 21394 consensus_queue.cc:237] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.425925 21393 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.427675 21389 consensus_queue.cc:1046] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250831 13:58:12.408944 21344 ts_tablet_manager.cc:1397] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Time spent bootstrapping tablet: real 0.359s user 0.040s sys 0.015s
I20250831 13:58:12.437647 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.438901 20467 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.444454 21392 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.445926 21389 raft_consensus.cc:3053] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.446317 21389 raft_consensus.cc:738] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:12.447273 21389 consensus_queue.cc:260] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:12.448714 21389 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.450152 21395 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.452610 21375 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.454838 21405 consensus_queue.cc:1046] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:12.456046 21405 raft_consensus.cc:3053] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.456408 21405 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.457567 21405 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.461925 21405 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.471081 20493 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:12.476837 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.478644 20467 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:12.480934 21396 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.481789 21377 consensus_queue.cc:1046] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:12.483044 21377 raft_consensus.cc:3053] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:12.483441 21377 raft_consensus.cc:738] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:12.484429 21377 consensus_queue.cc:260] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.489504 21399 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.489656 21377 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:12.475334 21384 consensus_queue.cc:1162] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "212b66976f874aa682cae0fb77928786" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20250831 13:58:12.472673 21380 consensus_queue.cc:260] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.501503 21380 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.513257 21404 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.514892 21344 raft_consensus.cc:357] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:12.516254 21344 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Initialized, Role: LEARNER
I20250831 13:58:12.517199 21344 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:12.525475 20572 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 13:58:12.527245 21391 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 13:58:12.582145 20501 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
W20250831 13:58:12.601964 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 235914us
I20250831 13:58:12.602838 20543 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.604123 20543 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:12.604674 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 46358.9us
I20250831 13:58:12.605126 20543 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.606578 20400 catalog_manager.cc:5582] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:12.608426 20400 catalog_manager.cc:5582] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:12.605857 21390 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.609601 21387 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.609503 21379 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.633962 21208 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 13:58:12.637421 21377 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 13:58:12.640262 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.642549 21390 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.645623 20543 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.647976 20543 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.649480 20543 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.647262 21344 ts_tablet_manager.cc:1428] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Time spent starting tablet: real 0.209s user 0.005s sys 0.005s
I20250831 13:58:12.658840 21379 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.652594 21390 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.648938 21412 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.661816 20572 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:12.662185 20496 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0"
dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52966
I20250831 13:58:12.662932 20496 raft_consensus.cc:604] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Received request to transfer leadership
I20250831 13:58:12.651742 20543 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.665583 21412 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.665241 20543 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.670020 20543 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.670519 21390 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.677809 21379 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.679890 20661 tablet_copy_service.cc:342] P 44d163094add49049cc749547d891602: Request end of tablet copy session ab7ade51fa134a1b814fab748f01d43d-2c47fb589f98457ab81d63e6b9d19fd0 received from {username='slave'} at 127.0.0.1:46464
I20250831 13:58:12.680368 20661 tablet_copy_service.cc:434] P 44d163094add49049cc749547d891602: ending tablet copy session ab7ade51fa134a1b814fab748f01d43d-2c47fb589f98457ab81d63e6b9d19fd0 on tablet 2c47fb589f98457ab81d63e6b9d19fd0 with peer ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:12.681170 20543 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.671940 20400 catalog_manager.cc:5582] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:12.684610 21379 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.686951 20543 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.688910 20543 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.699266 20543 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.700903 20543 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:12.702857 21387 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.698215 21379 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.699913 21390 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.707226 21387 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:12.735994 20572 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250831 13:58:12.737859 21402 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:12.777269 20572 raft_consensus.cc:1273] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:12.779508 21389 consensus_queue.cc:1035] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250831 13:58:12.779801 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 74334.5us
I20250831 13:58:12.780815 20543 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:12.790654 20501 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0"
dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52966
I20250831 13:58:12.791364 20501 raft_consensus.cc:604] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Received request to transfer leadership
W20250831 13:58:12.792763 20387 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 2c47fb589f98457ab81d63e6b9d19fd0 already in progress [suppressed 1 similar messages]
I20250831 13:58:12.784463 21412 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:12.822808 20501 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0"
dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52966
I20250831 13:58:12.823345 20501 raft_consensus.cc:604] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Received request to transfer leadership
W20250831 13:58:12.829746 20448 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 2c47fb589f98457ab81d63e6b9d19fd0 already in progress
W20250831 13:58:12.830087 20448 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 2c47fb589f98457ab81d63e6b9d19fd0 already in progress
I20250831 13:58:12.838922 21364 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 13:58:12.840219 21364 tablet_bootstrap.cc:492] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap complete.
I20250831 13:58:12.841085 21364 ts_tablet_manager.cc:1397] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Time spent bootstrapping tablet: real 0.603s user 0.024s sys 0.011s
I20250831 13:58:12.845218 20647 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250831 13:58:12.848572 21209 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Advancing to term 2
I20250831 13:58:12.855297 20647 raft_consensus.cc:1273] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:12.861546 21399 consensus_queue.cc:1035] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20250831 13:58:12.872929 21209 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 2 LEARNER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250831 13:58:12.861106 21389 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20250831 13:58:12.876137 21399 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:12.881013 21373 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 13:58:12.881916 21373 tablet_bootstrap.cc:492] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Bootstrap complete.
I20250831 13:58:12.882550 21373 ts_tablet_manager.cc:1397] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Time spent bootstrapping tablet: real 0.607s user 0.038s sys 0.040s
I20250831 13:58:12.887924 20571 raft_consensus.cc:1273] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:12.857235 21412 raft_consensus.cc:695] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:12.890044 21402 consensus_queue.cc:1035] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:12.913172 21412 consensus_queue.cc:237] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:12.906173 21402 raft_consensus.cc:991] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: : Instructing follower 212b66976f874aa682cae0fb77928786 to start an election
I20250831 13:58:12.926055 21389 raft_consensus.cc:1079] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Signalling peer 212b66976f874aa682cae0fb77928786 to start an election
I20250831 13:58:12.927903 21373 raft_consensus.cc:357] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:12.928735 21373 raft_consensus.cc:738] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Initialized, Role: LEARNER
I20250831 13:58:12.929373 21373 consensus_queue.cc:260] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:12.930331 20569 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0"
dest_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41400
I20250831 13:58:12.932279 21373 ts_tablet_manager.cc:1428] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Time spent starting tablet: real 0.049s user 0.004s sys 0.002s
I20250831 13:58:12.894848 21364 raft_consensus.cc:357] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:12.935850 21364 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Initialized, Role: LEARNER
I20250831 13:58:12.966219 21372 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250831 13:58:12.991596 21372 tablet_bootstrap.cc:492] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Bootstrap complete.
I20250831 13:58:12.992385 21372 ts_tablet_manager.cc:1397] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Time spent bootstrapping tablet: real 0.669s user 0.015s sys 0.025s
I20250831 13:58:13.028463 20511 tablet_copy_service.cc:342] P 58aded50ab0d4609a053647eb11b1f48: Request end of tablet copy session ab7ade51fa134a1b814fab748f01d43d-00c91580fd1b4c45a280862eedb8e1ac received from {username='slave'} at 127.0.0.1:37606
I20250831 13:58:13.029072 20511 tablet_copy_service.cc:434] P 58aded50ab0d4609a053647eb11b1f48: ending tablet copy session ab7ade51fa134a1b814fab748f01d43d-00c91580fd1b4c45a280862eedb8e1ac on tablet 00c91580fd1b4c45a280862eedb8e1ac with peer ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:13.009145 20569 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:58:13.012821 21389 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:12.984068 21364 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.043188 20569 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.048977 21389 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.089674 21208 raft_consensus.cc:1215] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 13:58:13.091143 21372 raft_consensus.cc:357] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.092036 21372 raft_consensus.cc:738] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Initialized, Role: LEARNER
I20250831 13:58:13.092730 21372 consensus_queue.cc:260] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.093859 21389 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.075616 20645 raft_consensus.cc:1273] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:13.097805 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.098639 21364 ts_tablet_manager.cc:1428] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Time spent starting tablet: real 0.257s user 0.009s sys 0.002s
I20250831 13:58:13.098598 20575 raft_consensus.cc:2391] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:13.099717 21372 ts_tablet_manager.cc:1428] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Time spent starting tablet: real 0.107s user 0.008s sys 0.002s
I20250831 13:58:13.104427 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.104976 20650 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.106784 20587 tablet_copy_service.cc:342] P 212b66976f874aa682cae0fb77928786: Request end of tablet copy session ab7ade51fa134a1b814fab748f01d43d-0c14599239fd4c99b3331692a513a092 received from {username='slave'} at 127.0.0.1:34436
I20250831 13:58:13.112532 20511 tablet_copy_service.cc:342] P 58aded50ab0d4609a053647eb11b1f48: Request end of tablet copy session ab7ade51fa134a1b814fab748f01d43d-90a0a56228d04660b3cb351485be6593 received from {username='slave'} at 127.0.0.1:37606
I20250831 13:58:13.112958 20511 tablet_copy_service.cc:434] P 58aded50ab0d4609a053647eb11b1f48: ending tablet copy session ab7ade51fa134a1b814fab748f01d43d-90a0a56228d04660b3cb351485be6593 on tablet 90a0a56228d04660b3cb351485be6593 with peer ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:13.107236 20587 tablet_copy_service.cc:434] P 212b66976f874aa682cae0fb77928786: ending tablet copy session ab7ade51fa134a1b814fab748f01d43d-0c14599239fd4c99b3331692a513a092 on tablet 0c14599239fd4c99b3331692a513a092 with peer ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:13.115962 20569 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.107846 21384 raft_consensus.cc:1062] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:13.118960 20400 catalog_manager.cc:5582] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 2, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
W20250831 13:58:13.117956 21384 raft_consensus.cc:1066] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Unable to promote non-voter ab7ade51fa134a1b814fab748f01d43d: Service unavailable: leader transfer in progress
I20250831 13:58:13.126032 21399 consensus_queue.cc:1035] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250831 13:58:13.129655 20466 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:13.135535 21408 raft_consensus.cc:1062] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
W20250831 13:58:13.147513 21408 raft_consensus.cc:1066] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Unable to promote non-voter ab7ade51fa134a1b814fab748f01d43d: Service unavailable: leader transfer in progress
I20250831 13:58:13.147940 21384 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:13.163334 21384 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.163825 21384 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:13.180138 20569 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.180987 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.183744 20645 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.182053 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.186738 20494 raft_consensus.cc:3053] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:13.187093 20494 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:13.187835 20494 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.189350 20494 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.191143 20645 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:13.193150 21393 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.193614 21393 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.196197 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.196801 20575 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.196995 20541 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:13.197783 21432 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.198225 21432 raft_consensus.cc:695] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:13.199294 20467 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.199070 21432 consensus_queue.cc:237] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.200035 21404 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.200373 21404 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.200731 21404 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.205021 20494 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:13.207638 21393 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.211001 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.213003 21404 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.214039 20647 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.219815 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.220355 20569 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.222796 21404 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.224979 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.225564 20645 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.228288 20396 catalog_manager.cc:5582] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 2 to 3, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:13.249323 20569 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:13.224592 21384 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.250344 20467 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.251273 21404 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.251842 21404 raft_consensus.cc:695] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:13.252671 21404 consensus_queue.cc:237] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.253960 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.254019 21393 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.254702 21393 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:13.259694 21393 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:13.260843 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:13.261447 21209 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.262436 20468 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:13.263211 21393 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.263605 21393 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.263988 21393 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.265043 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.265641 20647 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:13.267390 21384 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.269131 20650 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.270175 20466 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.271359 20645 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:13.272958 21384 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:13.274107 21384 raft_consensus.cc:695] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:13.273296 21393 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:13.277606 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.278318 20647 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.280220 21393 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:13.280854 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:13.281455 21209 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.283128 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.283882 20569 raft_consensus.cc:2391] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:13.286084 21384 consensus_queue.cc:237] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.288378 21209 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:13.289367 20468 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:13.290212 21393 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.290728 21393 raft_consensus.cc:695] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:13.291589 21393 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:13.305737 21399 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.306272 21399 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.308516 21399 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.309098 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.309926 20575 raft_consensus.cc:2391] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:13.310989 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.311583 20650 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.302121 20396 catalog_manager.cc:5582] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:13.314059 20466 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:13.314792 21399 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:13.315210 21399 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.315662 21209 raft_consensus.cc:1215] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250831 13:58:13.317534 20647 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:13.315660 21399 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:13.332386 21399 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.334831 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.335497 20575 raft_consensus.cc:2391] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:13.342283 21399 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.342811 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.351569 20645 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.352730 20466 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:13.353893 21393 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:13.354441 21393 raft_consensus.cc:695] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:13.355880 21393 consensus_queue.cc:237] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.358798 20400 catalog_manager.cc:5582] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:13.360548 20400 catalog_manager.cc:5582] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:13.386245 21393 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.386793 21393 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.389591 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.389655 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.390211 20645 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.390307 20575 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.391414 20466 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.392113 21393 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.392351 21384 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.392827 21384 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.385367 20400 catalog_manager.cc:5582] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:13.393165 21384 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.405786 21384 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.408311 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.408717 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.408936 20575 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.409368 20645 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.413426 20647 raft_consensus.cc:1273] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:13.414839 21432 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:13.425593 20575 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:13.427004 21418 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:13.427657 21418 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.428992 21430 raft_consensus.cc:1062] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:13.431582 20467 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.432971 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.433522 20501 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:13.433615 21404 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.434753 21404 raft_consensus.cc:695] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:13.435072 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.435756 20648 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:13.435588 21404 consensus_queue.cc:237] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.437529 21384 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.417426 20645 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:13.435499 21430 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:13.438267 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.440145 21432 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:13.440514 21432 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:13.440914 21432 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:13.430083 21418 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.458047 21432 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.459389 20501 raft_consensus.cc:1273] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:13.461213 21436 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:13.463074 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.464306 21432 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.465015 21377 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.465754 21377 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.470248 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.471027 20496 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:13.474071 20617 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.475153 21391 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.475620 21391 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.475986 21391 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.478964 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.480962 21349 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.481555 21209 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:13.483165 21430 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:13.479593 20650 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:13.481424 21349 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.493739 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.494385 20569 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.495329 21349 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.501360 21404 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.501945 21404 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.503816 21380 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.504272 21380 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.504863 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.506445 20496 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:13.504261 21404 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.506394 21380 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.508258 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.509397 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.509595 20496 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.507227 21430 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.510375 21418 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:13.510650 21430 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.510983 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.511603 20569 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.511356 20400 catalog_manager.cc:5582] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:13.512581 21430 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.513063 20617 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.513836 21380 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.510025 20645 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.514227 21380 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.514751 21380 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.515154 20466 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.515889 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.490523 20501 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:13.516463 21384 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.517043 21384 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.516502 20575 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.517433 21384 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.519745 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.520988 20577 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.521154 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.520335 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.468204 21377 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.521907 20496 raft_consensus.cc:2391] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:13.522361 21432 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:13.522886 21432 raft_consensus.cc:695] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:13.523754 21432 consensus_queue.cc:237] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.526432 20616 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:13.527824 21405 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.528277 21405 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.528677 21405 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.538594 21384 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.547418 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.548161 20577 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.551420 21349 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.551908 21349 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.553804 21391 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.555956 20577 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:13.556553 21349 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.557209 20467 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.558365 21404 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.558916 21404 raft_consensus.cc:695] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:13.561002 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.561553 20571 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:13.543880 21380 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.563246 20650 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:13.562003 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.565366 20497 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.566254 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.566551 20617 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.566912 20650 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:13.569568 20541 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:13.571779 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.572309 21436 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:13.572309 20650 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.574393 21377 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.574822 21377 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.575158 21377 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.576706 21430 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:13.577073 21430 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.577400 21430 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:13.578845 21384 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.582842 20650 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:13.584920 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.585721 20496 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.587443 21377 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.590921 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.594056 20650 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250831 13:58:13.580271 21405 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.588896 21377 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.592182 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.599037 20494 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.595937 21404 consensus_queue.cc:237] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.607887 21380 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.608356 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.608901 20499 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.611883 21384 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.612442 21384 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.614420 21384 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.615144 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.616150 20577 raft_consensus.cc:2391] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:13.616263 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.617786 20650 raft_consensus.cc:2366] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3.
I20250831 13:58:13.619244 20466 leader_election.cc:400] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3.
I20250831 13:58:13.619547 20466 leader_election.cc:403] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:13.620530 21384 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.615095 20499 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.626065 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.626950 20577 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.627417 20617 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.635353 21412 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:13.636307 20494 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.638106 21405 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.638484 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.638890 21380 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.639060 20575 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.639415 21380 raft_consensus.cc:695] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:13.639532 20617 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.640306 21380 consensus_queue.cc:237] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.640916 21375 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.641373 21375 raft_consensus.cc:695] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:13.642179 21375 consensus_queue.cc:237] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.644496 21391 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.648288 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.648108 21349 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.650944 20575 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.616776 20496 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.642194 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.655061 20569 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.647588 20577 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.669788 20617 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.672821 21460 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.673382 21460 raft_consensus.cc:695] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:13.674396 21460 consensus_queue.cc:237] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.656157 21349 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.706586 20569 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.706419 20396 catalog_manager.cc:5582] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 2, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:13.710753 20501 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250831 13:58:13.708217 21384 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3.
I20250831 13:58:13.725632 21418 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.733680 20616 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:13.733271 21375 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.717489 21457 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.736188 21457 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:13.736979 21458 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.737435 21458 raft_consensus.cc:695] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:13.738163 21457 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:13.740260 21458 consensus_queue.cc:237] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.742096 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:13.742759 21208 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.729861 21404 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.743803 21404 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:13.736083 21375 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.745692 21404 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:13.746621 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.746631 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:13.747025 21349 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.747502 21208 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:13.748338 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.748987 20496 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.746894 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.748916 20468 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:13.749643 20501 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.697783 21430 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.748337 20617 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:13.756353 21405 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.756728 21405 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.757081 21405 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.757871 20571 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.707412 21431 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:13.758816 21418 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.759999 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:13.757103 21430 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.762697 20617 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.763345 21404 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.763639 20617 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.763888 21404 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.764047 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.764825 20496 raft_consensus.cc:2366] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:13.764668 21391 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.766041 21391 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.766362 21391 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.767990 20571 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.770242 21418 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.772228 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.772672 21377 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.773133 21377 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:13.773609 21377 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.753262 21375 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.717408 21436 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20250831 13:58:13.730029 21425 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.774955 21425 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.755262 20645 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:13.773066 21405 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:13.779738 20543 leader_election.cc:400] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:13.783231 20543 leader_election.cc:403] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250831 13:58:13.782073 20400 catalog_manager.cc:5582] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:13.780844 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:13.752983 20493 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:13.780089 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.789177 20501 raft_consensus.cc:2391] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:13.794929 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:13.795665 20497 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:13.798171 20494 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.777051 21425 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:13.790122 21430 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:13.799896 21391 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.801900 21391 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.798214 21405 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:13.802536 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.803158 20501 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.803278 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:13.804163 21209 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.809849 21393 raft_consensus.cc:1062] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:13.809728 20501 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.772992 20647 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:13.812250 20617 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:13.812752 21208 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 2 LEARNER]: Advancing to term 3
I20250831 13:58:13.812975 21405 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:13.813475 21405 raft_consensus.cc:695] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:13.814352 21405 consensus_queue.cc:237] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:13.783902 21404 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.817021 21393 consensus_queue.cc:237] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:13.895421 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 77135.9us
I20250831 13:58:13.893224 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:13.903010 20396 catalog_manager.cc:5582] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:13.906399 20396 catalog_manager.cc:5582] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:13.912498 21208 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 3 LEARNER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250831 13:58:13.912278 21377 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.915609 20648 raft_consensus.cc:2366] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
W20250831 13:58:13.918800 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 61506.8us
I20250831 13:58:13.921059 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.921627 20571 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:13.921725 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.922544 20575 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.927125 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.929437 20570 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.935989 21430 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:13.938786 20572 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:13.925611 21418 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:13.941135 21418 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:13.941632 21418 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:13.926242 21466 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:13.918183 21412 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:13.950958 21466 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:13.951098 21412 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:13.953032 21466 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:13.957433 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.957830 21209 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.958678 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:13.959277 20575 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:13.958030 20577 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.953065 21412 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:13.966734 20616 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:13.960883 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:13.976384 20576 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:13.976784 20541 leader_election.cc:400] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:13.977202 20541 leader_election.cc:403] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:13.978312 20617 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:13.979542 20570 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.978546 20396 catalog_manager.cc:5582] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 reported cstate change: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:58:13.993266 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:13.993857 20501 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.004021 20501 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:13.992362 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.007997 20498 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:13.978663 21459 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:13.980647 21431 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:14.005723 21466 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:14.005694 21349 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:14.023573 21436 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.025920 21467 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:14.017207 21459 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:14.021510 21431 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.033844 20650 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:14.036378 21431 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:14.036836 20648 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.037542 21412 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:14.038026 21412 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.039920 21412 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:14.030581 21466 raft_consensus.cc:695] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:14.041383 21466 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:14.044842 20650 raft_consensus.cc:1273] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.050760 20650 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250831 13:58:14.051669 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 134600us
I20250831 13:58:14.054019 21436 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:14.028805 21377 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.031265 21349 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.054723 21209 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:14.056912 21349 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.059408 21418 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.061834 21418 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:14.033303 21459 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.081638 20576 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:14.079144 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:13.933804 20647 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:14.126205 21472 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:14.126889 21460 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:14.127460 21460 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.129568 21460 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.131667 21384 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:14.123351 21377 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:14.157228 21208 raft_consensus.cc:2391] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
I20250831 13:58:14.156872 21404 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:14.160451 21459 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.136946 21377 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.180734 20576 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.182754 20576 raft_consensus.cc:1273] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.181993 21479 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:14.185766 20577 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:14.190744 21479 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.193203 21479 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:14.195883 21377 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.196650 21459 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.185554 21464 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:14.198999 21464 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.199597 20617 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:14.200522 21460 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:14.200953 21464 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.201304 21460 raft_consensus.cc:695] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:14.202651 21404 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:14.202239 21460 consensus_queue.cc:237] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.204855 21209 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:14.207995 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.208571 20498 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:14.209036 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.209678 20499 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:14.211189 20501 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.214466 21404 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:14.215008 20493 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.218858 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:14.219995 20617 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:14.232642 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:14.233372 21209 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.239256 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.239974 20497 raft_consensus.cc:2366] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:14.240728 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.241484 20493 raft_consensus.cc:2391] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:14.242208 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:14.242903 20495 raft_consensus.cc:2391] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:14.244978 20497 raft_consensus.cc:1273] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:14.246374 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.246909 21485 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:14.247424 20501 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:14.250524 21485 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:14.252418 21404 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:14.261791 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.264106 20499 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:14.263262 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:14.268306 20501 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.273258 21209 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:14.274150 21393 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:14.275424 21207 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:14.274320 21466 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20250831 13:58:14.278220 20617 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:14.274277 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.279265 20500 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:14.267769 21458 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:14.286298 21458 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:14.286746 21458 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:14.289897 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 315041us
I20250831 13:58:14.260851 21464 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:14.291404 21464 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:14.291859 21464 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:14.299891 21459 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:14.300319 21459 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:14.300664 21459 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.302860 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:14.274183 21463 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20250831 13:58:14.305608 20617 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:14.313336 20500 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.316737 20576 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:14.317518 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:14.318331 20572 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:14.321593 20570 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.322185 20571 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.323364 20468 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:14.324141 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:14.325098 20572 raft_consensus.cc:2391] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:14.326100 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:14.326663 20571 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:14.328718 20543 leader_election.cc:400] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:14.329082 20543 leader_election.cc:403] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:14.331163 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:14.331851 20573 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.336458 20575 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:14.338006 20501 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:14.339648 20617 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:14.342130 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:14.348940 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:14.351385 21384 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:14.352703 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:14.361486 20570 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.339745 21466 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:14.346493 21468 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20250831 13:58:14.346638 21349 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:14.344475 21467 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:14.355547 21473 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:14.363457 20501 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:14.412071 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:14.429368 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 357744us
I20250831 13:58:14.433019 21399 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:14.431852 21467 raft_consensus.cc:695] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:14.437922 20500 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 13:58:14.425158 20568 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:14.500167 20568 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:14.361317 21384 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:14.352478 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:14.494133 20575 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:14.339046 21457 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250831 13:58:14.364099 21466 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:14.509089 20573 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:14.515280 20570 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:14.518846 21418 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
W20250831 13:58:14.508610 21449 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20250831 13:58:14.530891 21479 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:14.501418 21349 raft_consensus.cc:695] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:14.509038 21466 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.532370 21404 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:14.523389 21476 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:14.500861 21467 consensus_queue.cc:237] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:14.507299 21384 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:14.578713 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 143699us
I20250831 13:58:14.622984 21349 consensus_queue.cc:237] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.560111 21405 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250831 13:58:14.663357 21209 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250831 13:58:14.658445 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:14.575310 21476 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.673599 21476 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:14.673192 20396 catalog_manager.cc:5582] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:14.652143 21436 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20250831 13:58:14.679729 21375 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:14.675199 20396 catalog_manager.cc:5582] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:14.686482 21207 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 1 LEARNER]: Advancing to term 3
I20250831 13:58:14.681849 20396 catalog_manager.cc:5582] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:14.702976 21436 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:14.703702 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:14.703776 21458 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.705273 20647 raft_consensus.cc:2366] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:14.710516 21459 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.711822 20649 raft_consensus.cc:1238] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:14.725788 21459 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.726564 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:14.732218 20501 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:14.739212 20543 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:14.747536 21431 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:14.760391 21431 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:14.760852 21431 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.662168 21464 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.741007 20646 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:14.800634 21207 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 3 LEARNER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:14.802430 21493 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:14.803664 21464 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.805436 21207 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 44d163094add49049cc749547d891602 (127.18.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:14.810256 21458 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.812276 21466 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.814726 20651 raft_consensus.cc:2366] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:14.816876 21431 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.817332 21384 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.824754 21384 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:14.819073 21432 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 44d163094add49049cc749547d891602 (127.18.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:14.822734 21466 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:14.865679 21431 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:14.924607 21491 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:14.935463 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:14.936707 20493 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:14.934427 20495 raft_consensus.cc:2953] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 44d163094add49049cc749547d891602 (127.18.233.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:14.925843 21491 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:14.949309 21491 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:14.951673 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:15.011591 20614 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.002s sys 0.061s
W20250831 13:58:15.002357 20499 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.010s sys 0.058s
W20250831 13:58:15.027062 20614 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s user 0.003s sys 0.061s
W20250831 13:58:15.027493 20499 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s user 0.010s sys 0.058s
W20250831 13:58:15.037046 21206 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.005s sys 0.026s
W20250831 13:58:15.045408 20541 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.031s sys 0.047s
W20250831 13:58:15.052528 20541 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.031s sys 0.047s
W20250831 13:58:15.047472 20542 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.039s sys 0.035s
W20250831 13:58:15.049274 21206 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.005s sys 0.027s
W20250831 13:58:15.054634 20542 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.039s sys 0.035s
W20250831 13:58:15.056918 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 609800us
W20250831 13:58:15.054513 20466 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s user 0.008s sys 0.019s
I20250831 13:58:15.059327 20493 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
W20250831 13:58:15.063982 20615 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s user 0.022s sys 0.002s
W20250831 13:58:15.072633 20615 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s user 0.022s sys 0.002s
I20250831 13:58:15.081275 21466 raft_consensus.cc:1062] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
W20250831 13:58:15.081821 21466 raft_consensus.cc:1066] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Unable to promote non-voter ab7ade51fa134a1b814fab748f01d43d: Illegal state: Leader has not yet committed an operation in its own term
I20250831 13:58:15.089756 21384 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250831 13:58:15.138270 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 464331us
I20250831 13:58:15.162365 20494 raft_consensus.cc:1273] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:15.180287 20642 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:15.197119 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:15.198030 20571 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:15.201488 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:15.201921 21375 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:15.202288 20501 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:15.202587 21375 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.205003 21375 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.205137 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:15.205483 21375 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:15.205772 20502 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:15.207007 21464 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:15.207792 21464 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.206139 21375 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.211998 20541 leader_election.cc:400] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:15.219907 20541 leader_election.cc:403] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:15.221786 20571 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:15.252355 20502 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:15.201570 21466 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:15.254374 21466 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.272949 20499 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:15.213552 21502 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:15.302937 20501 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:15.277287 21466 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.212558 21459 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.161s
I20250831 13:58:15.373034 21464 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.407395 21432 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:15.408622 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:15.409463 20648 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:15.409883 20543 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:15.411868 21467 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:15.412441 21431 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:15.413012 21431 raft_consensus.cc:695] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:15.424129 20648 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:15.414057 21431 consensus_queue.cc:237] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.395594 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:15.437500 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 225764us
I20250831 13:58:15.382095 21502 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:15.456728 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:15.457418 20648 raft_consensus.cc:2374] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3. Re-sending same reply.
I20250831 13:58:15.439846 21206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47746 (request call id 24) took 1344 ms. Trace:
I20250831 13:58:15.491346 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:15.498787 21496 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:15.499027 21464 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:15.499497 21496 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.499867 21464 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.501960 21496 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.502226 21464 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.503391 21209 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:15.509119 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:15.490267 21349 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:15.508769 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:15.480717 21375 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.535220 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:15.513252 21432 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:15.478561 20642 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
W20250831 13:58:15.549103 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 137595us
I20250831 13:58:15.516091 21502 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:15.553251 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:15.553820 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:15.562119 20649 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:15.562254 21511 log.cc:927] Time spent T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250831 13:58:15.570654 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:15.571772 21465 raft_consensus.cc:2762] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20250831 13:58:15.507618 21473 consensus_queue.cc:1046] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20250831 13:58:15.565012 21375 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:15.593894 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:15.587528 21459 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:15.566327 21349 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.611989 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:15.615155 21493 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20250831 13:58:15.601675 21465 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:15.617681 21465 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:15.619721 21465 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:15.620235 21509 raft_consensus.cc:1062] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
W20250831 13:58:15.620723 21509 raft_consensus.cc:1066] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Unable to promote non-voter ab7ade51fa134a1b814fab748f01d43d: Illegal state: Leader has not yet committed an operation in its own term
W20250831 13:58:15.620837 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 89018.5us
I20250831 13:58:15.621709 20541 leader_election.cc:400] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:15.621974 20541 leader_election.cc:403] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:15.624181 20541 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:15.624902 21432 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:15.607398 21473 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:15.630244 21502 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:15.703862 20498 raft_consensus.cc:1273] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:15.611310 21375 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:15.708679 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 68577.9us
I20250831 13:58:15.651137 21508 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:15.711743 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 86058.2us
W20250831 13:58:15.710090 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 110453us
I20250831 13:58:15.640895 21459 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.709853 21473 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.768448 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:58:15.773824 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 76607us
I20250831 13:58:15.736280 21508 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
W20250831 13:58:15.804653 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 152183us
I20250831 13:58:15.765759 21457 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.137s
W20250831 13:58:15.802337 21527 log.cc:927] Time spent T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20250831 13:58:15.802944 21537 log.cc:927] Time spent T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20250831 13:58:15.491004 21206 rpcz_store.cc:276] 0831 13:58:14.090008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:14.090163 (+ 155us) service_pool.cc:225] Handling call
0831 13:58:14.090542 (+ 379us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:14.259568 (+169026us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:14.266947 (+ 7379us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:14.331897 (+ 64950us) log.cc:844] Serialized 26 byte log entry
0831 13:58:15.081410 (+749513us) spinlock_profiling.cc:243] Waited 961 us on lock 0x7b1800153ca0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9bc5c255c 00007fb9b99f777b 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171
0831 13:58:15.081746 (+ 336us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:15.345832 (+264086us) log.cc:844] Serialized 14 byte log entry
0831 13:58:15.373344 (+ 27512us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:15.382567 (+ 9223us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:15.384958 (+ 2391us) raft_consensus.cc:1683] finished
0831 13:58:15.404838 (+ 19880us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:15.434507 (+ 29669us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":354107,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":1057,"raft.queue_time_us":262196,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":31982,"spinlock_wait_cycles":2209152,"thread_start_us":1067178,"threads_started":3,"wal-append.queue_time_us":709863}
W20250831 13:58:15.803808 21526 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
I20250831 13:58:15.859911 20617 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:15.792526 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:15.805804 21349 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.858835 20649 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:15.886713 21483 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:15.887372 21483 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:15.888083 21508 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.889571 21483 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:15.890221 20645 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250831 13:58:15.890933 21544 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:15.891515 21544 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:15.894214 21544 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:15.880825 21531 log.cc:927] Time spent T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
W20250831 13:58:15.898372 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 184322us
I20250831 13:58:15.879949 21375 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:15.928344 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 146397us
W20250831 13:58:15.960744 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 107917us
W20250831 13:58:15.950721 21530 log.cc:927] Time spent T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
I20250831 13:58:15.969758 20617 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:15.916783 21508 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:15.927783 21459 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.951551 21412 raft_consensus.cc:3053] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:15.939107 21547 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:15.951227 21546 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:15.987203 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 224158us
I20250831 13:58:15.976539 21473 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:15.990589 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:15.992497 20646 raft_consensus.cc:1238] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:15.978256 21496 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:15.994822 21375 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:15.995306 21375 raft_consensus.cc:695] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:15.995554 20386 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0c14599239fd4c99b3331692a513a092 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250831 13:58:15.994419 21496 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.996366 21375 consensus_queue.cc:237] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:15.998315 21496 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:15.998976 21491 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:15.999845 21552 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:15.999516 21547 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.000483 21552 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.001050 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:15.999616 21545 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:15.999567 21491 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.007768 21552 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:16.004706 21532 log.cc:927] Time spent T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.210s user 0.000s sys 0.001s
W20250831 13:58:16.011147 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 46714.1us
I20250831 13:58:15.979897 21431 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:15.985460 21508 raft_consensus.cc:695] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:15.983464 21412 raft_consensus.cc:738] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:16.037946 21555 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:16.038341 20569 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:16.045969 21546 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.040176 20577 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:16.038620 21555 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.004106 21554 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:16.043911 21540 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:16.038476 21412 consensus_queue.cc:260] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.038035 21545 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.050998 21431 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.058990 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:16.060384 21545 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.023857 21464 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20250831 13:58:16.060890 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:16.061797 21555 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.062045 21508 consensus_queue.cc:237] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:16.066361 21539 log.cc:927] Time spent T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.300s user 0.001s sys 0.000s
I20250831 13:58:16.066833 21556 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:16.067641 21412 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:16.036226 21473 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.043772 21491 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.044246 20572 raft_consensus.cc:1273] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:16.047408 21432 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:16.068122 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.095456 21564 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:16.053841 21552 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.098485 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:16.122084 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 270931us
W20250831 13:58:16.125597 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 90160.7us
I20250831 13:58:16.126477 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.068763 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:16.115808 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:16.068182 21546 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:16.138162 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44234.7us
I20250831 13:58:16.095785 21547 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:16.138717 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.107025 21556 raft_consensus.cc:695] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:16.143500 21552 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.147979 21551 raft_consensus.cc:2953] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:16.171027 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.136677 21491 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.178838 21503 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.180351 21491 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.182615 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.182432 21503 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.183923 21491 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.184198 20569 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:16.095091 21540 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:16.173158 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:16.180903 20493 raft_consensus.cc:2366] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:16.198004 21431 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.200100 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.202016 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.202526 21497 raft_consensus.cc:1062] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:16.203123 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.203644 21556 consensus_queue.cc:237] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.095741 21554 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:16.226900 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.227977 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.229542 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.185483 21503 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.157482 21473 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.239584 21473 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.242619 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.220335 20576 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:16.140756 21564 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.249871 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.250162 20541 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:16.250634 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.245152 20616 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:16.235157 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 790) took 1032 ms. Trace:
I20250831 13:58:16.237211 21431 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:16.259435 20616 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:16.260948 20616 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:16.262217 20616 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:16.263697 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.264115 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.264811 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.267961 21565 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:16.268517 21565 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.269481 21431 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.257081 20493 rpcz_store.cc:276] 0831 13:58:15.200189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:15.204635 (+ 4446us) service_pool.cc:225] Handling call
0831 13:58:16.115714 (+911079us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d766c 00007fb9b99d75d6 000000000035a3c9 00007fb9c185e300 00007fb9c1924ec4 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:16.232173 (+116459us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23870848}
I20250831 13:58:16.270931 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.271566 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.272759 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.272845 21496 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:16.273677 21496 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:16.275297 20568 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:16.286864 20643 raft_consensus.cc:1273] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:16.290864 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:16.285035 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 83799.5us
I20250831 13:58:16.277845 21209 raft_consensus.cc:2953] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:16.296008 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.279012 21544 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:16.296301 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.246476 21509 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20250831 13:58:16.281137 21503 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.287500 21457 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:16.252681 21464 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250831 13:58:16.250626 21375 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20250831 13:58:16.288010 21432 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.287870 21545 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:16.309437 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.305392 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.294098 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.290015 21349 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20250831 13:58:16.310635 20576 raft_consensus.cc:1273] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:16.437062 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.825304 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.825309 21549 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.794884 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.826149 21549 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:16.673458 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 232569us
I20250831 13:58:16.453132 21552 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.828346 21549 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.828773 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.829026 21549 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.829669 21549 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.830399 20495 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:16.293426 21565 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.830024 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.831437 21565 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:16.831692 21549 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.831532 20499 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:16.832203 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.836970 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 655) took 1637 ms. Trace:
I20250831 13:58:16.832180 21565 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.837087 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.837376 20645 rpcz_store.cc:276] 0831 13:58:15.199871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:15.201044 (+ 1173us) service_pool.cc:225] Handling call
0831 13:58:15.574764 (+373720us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:15.575014 (+ 250us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:15.890601 (+315587us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:16.836721 (+946120us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54001eee54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:16.836881 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":660793088}
I20250831 13:58:16.838893 21565 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.839673 21565 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:16.840214 21565 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.827903 21564 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.840919 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:16.848570 20617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.014s sys 0.102s
W20250831 13:58:16.848999 20617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.014s sys 0.102s
W20250831 13:58:16.666435 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 381738us
I20250831 13:58:16.715977 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.323573 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:16.665234 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 364951us
W20250831 13:58:16.820294 21569 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.514s user 0.001s sys 0.000s
W20250831 13:58:16.521322 21570 log.cc:927] Time spent T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.241s user 0.001s sys 0.000s
I20250831 13:58:16.551514 21431 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.865988 20396 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59942 (request call id 145) took 2590 ms. Trace:
I20250831 13:58:16.453714 21465 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:16.436553 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:58:16.421918 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 116566us
I20250831 13:58:16.859356 21552 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:16.669970 21554 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:16.772464 21540 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:16.466747 21546 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:16.860262 21206 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.006s sys 0.007s
I20250831 13:58:16.311033 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.668373 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.448529 21503 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.879480 20576 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:16.474334 21566 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.632s
W20250831 13:58:16.890980 21206 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s user 0.006s sys 0.007s
I20250831 13:58:16.875584 21545 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:16.944743 21412 consensus_meta.cc:287] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.877s user 0.004s sys 0.006s
I20250831 13:58:16.945863 20577 raft_consensus.cc:2408] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20250831 13:58:16.947507 20575 consensus_meta.cc:287] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.728s user 0.008s sys 0.000s
I20250831 13:58:16.948128 20575 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:16.949065 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:16.896112 21498 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.901s user 0.003s sys 0.002s
I20250831 13:58:16.877789 21465 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.903596 21564 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:16.964908 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.968281 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.968853 20576 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:16.969275 21564 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:16.972402 20616 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:16.972872 20570 raft_consensus.cc:2953] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:16.975636 21466 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:16.976178 21466 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.951823 21584 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20250831 13:58:16.978416 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:16.979794 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:16.891527 21552 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.863356 21432 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.992515 20575 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:17.017243 21558 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:17.018136 21558 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.969306 21465 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:16.964545 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.000834 21564 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:16.960947 21546 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:16.962234 21545 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.100239 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:17.082635 20543 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.020s sys 0.088s
I20250831 13:58:17.089766 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:17.092913 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.104579 20576 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
W20250831 13:58:17.084764 21209 consensus_meta.cc:287] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.781s user 0.005s sys 0.005s
W20250831 13:58:17.048564 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 55170.4us
I20250831 13:58:17.052825 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:17.103456 20543 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s user 0.023s sys 0.088s
W20250831 13:58:17.113843 20541 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.815s user 0.011s sys 0.008s
W20250831 13:58:17.038722 21578 log.cc:927] Time spent T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.318s user 0.000s sys 0.004s
W20250831 13:58:17.155442 20541 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.857s user 0.011s sys 0.010s
W20250831 13:58:17.049840 21501 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.009s sys 0.017s
I20250831 13:58:17.056228 21565 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.158609 21560 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.026815 21558 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:17.179293 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 234394us
I20250831 13:58:17.162771 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:17.043423 21176 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:17.093081 21528 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.050035 21494 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.160354 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.038414 21582 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:17.051846 21432 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.023627 21466 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:17.021216 21586 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:17.198215 21558 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:17.180233 21501 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.874s user 0.013s sys 0.020s
I20250831 13:58:17.200896 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 655) took 1046 ms. Trace:
I20250831 13:58:17.201953 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.180991 20576 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:17.182366 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:16.870723 20396 rpcz_store.cc:276] 0831 13:58:14.233603 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:14.237880 (+ 4277us) service_pool.cc:225] Handling call
0831 13:58:16.824144 (+2586264us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:15.095828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:15.096916 (+ 1088us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:15.096986 (+ 70us) write_op.cc:435] Acquired schema lock
0831 13:58:15.097062 (+ 76us) tablet.cc:660] Decoding operations
0831 13:58:15.098222 (+ 1160us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:15.098344 (+ 122us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:15.098430 (+ 86us) tablet.cc:683] Acquiring locks for 3 operations
0831 13:58:15.099051 (+ 621us) tablet.cc:699] Row locks acquired
0831 13:58:15.099106 (+ 55us) write_op.cc:260] PREPARE: finished
0831 13:58:15.099417 (+ 311us) write_op.cc:270] Start()
0831 13:58:15.099697 (+ 280us) write_op.cc:276] Timestamp: P: 1756648695099229 usec, L: 0
0831 13:58:15.099757 (+ 60us) op_driver.cc:348] REPLICATION: starting
0831 13:58:15.101380 (+ 1623us) log.cc:844] Serialized 1384 byte log entry
0831 13:58:15.840462 (+739082us) op_driver.cc:464] REPLICATION: finished
0831 13:58:16.437329 (+596867us) write_op.cc:301] APPLY: starting
0831 13:58:16.447898 (+ 10569us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:16.506366 (+ 58468us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:16.540171 (+ 33805us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:16.540795 (+ 624us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:16.541039 (+ 244us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0831 13:58:16.541099 (+ 60us) write_op.cc:312] APPLY: finished
0831 13:58:16.541347 (+ 248us) log.cc:844] Serialized 34 byte log entry
0831 13:58:16.541549 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:16.541642 (+ 93us) write_op.cc:454] Released schema lock
0831 13:58:16.541691 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:16.542118 (+ 427us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:16.542143 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2982,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":440622,"apply.run_cpu_time_us":2790,"apply.run_wall_time_us":234201,"num_ops":3,"prepare.queue_time_us":344747,"prepare.run_cpu_time_us":32908,"prepare.run_wall_time_us":141784,"raft.queue_time_us":198062,"raft.run_cpu_time_us":4554,"raft.run_wall_time_us":257180,"replication_time_us":670691,"thread_start_us":602452,"threads_started":4,"wal-append.queue_time_us":419063}]]}
I20250831 13:58:17.055979 21581 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.191294 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:17.206645 20571 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:17.194912 21585 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.162101 21589 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:17.195613 21509 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.209939 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.197479 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.197571 21552 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.197688 21432 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.211977 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.208384 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:17.201340 20573 rpcz_store.cc:276] 0831 13:58:16.154534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.842617 (+688083us) service_pool.cc:225] Handling call
0831 13:58:16.842952 (+ 335us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:16.843132 (+ 180us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:17.014733 (+171601us) log.cc:844] Serialized 14 byte log entry
0831 13:58:17.200533 (+185800us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.200593 (+ 60us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:17.200645 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:17.200717 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:17.200835 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11269,"prepare.queue_time_us":325530,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":7649,"thread_start_us":345247,"threads_started":2}
I20250831 13:58:17.192397 21565 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.225304 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:17.191115 21566 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.214107 20617 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:17.195773 21584 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.213291 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 406) took 1153 ms. Trace:
I20250831 13:58:17.228902 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.209129 21501 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.233340 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.236171 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.213461 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 663) took 1735 ms. Trace:
I20250831 13:58:17.214850 21552 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.198752 21558 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.231711 20617 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:17.210628 21503 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.181890 21560 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.274190 20651 rpcz_store.cc:276] 0831 13:58:15.365670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:15.557580 (+191910us) service_pool.cc:225] Handling call
0831 13:58:15.557951 (+ 371us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:15.741775 (+183824us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:15.743644 (+ 1869us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:15.747676 (+ 4032us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:15.754600 (+ 6924us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:17.085944 (+1331344us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5400283054. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:17.088355 (+ 2411us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:17.100674 (+ 12319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":401730432}
I20250831 13:58:17.207780 21528 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.286556 20651 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57684
I20250831 13:58:17.287304 20651 raft_consensus.cc:604] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 LEADER]: Received request to transfer leadership
I20250831 13:58:17.206055 21586 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
W20250831 13:58:17.281528 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 49918.5us
I20250831 13:58:17.230325 21545 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.230157 21508 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.233831 21464 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:17.260417 21565 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:17.294376 21602 log.cc:927] Time spent T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20250831 13:58:17.294725 21544 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:17.228292 21432 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.277676 21501 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.284281 21552 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.301436 20617 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:17.283066 21551 consensus_meta.cc:287] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.989s user 0.012s sys 0.000s
I20250831 13:58:17.279989 21564 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:17.308499 21458 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:17.308660 21528 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.209829 21581 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:17.224890 21509 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.309171 20617 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:17.310756 21581 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.311070 21509 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:17.311564 21509 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:17.214059 21494 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.227648 21589 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.316339 21589 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.317013 21560 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:17.314746 21494 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:17.314724 20576 raft_consensus.cc:1273] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:17.273938 21603 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.236341 21582 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.285847 21554 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:17.323941 21590 log.cc:927] Time spent T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.232s user 0.001s sys 0.001s
I20250831 13:58:17.302301 21503 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.240710 21584 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:17.303463 21556 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:17.233490 21566 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:17.230005 21585 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.323709 21603 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.331660 21552 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:17.335208 21605 log.cc:927] Time spent T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.101s user 0.000s sys 0.000s
W20250831 13:58:17.335577 20465 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.000s sys 0.034s
W20250831 13:58:17.335898 20465 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s user 0.000s sys 0.034s
I20250831 13:58:17.341751 21556 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.346166 21599 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:17.352504 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 815) took 1067 ms. Trace:
I20250831 13:58:17.320354 21508 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.321630 21564 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.236011 20647 rpcz_store.cc:276] 0831 13:58:15.939330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:15.951929 (+ 12599us) service_pool.cc:225] Handling call
0831 13:58:15.977457 (+ 25528us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:16.000910 (+ 23453us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:16.000942 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:16.000990 (+ 48us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:16.001023 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:17.076008 (+1074985us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b540029fc54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:17.076073 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:17.092827 (+ 16754us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53428096}
I20250831 13:58:17.338528 21546 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.352991 20499 rpcz_store.cc:276] 0831 13:58:16.285317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.833291 (+547974us) service_pool.cc:225] Handling call
0831 13:58:16.833561 (+ 270us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:16.833858 (+ 297us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:16.833902 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:16.834458 (+ 556us) log.cc:844] Serialized 26 byte log entry
0831 13:58:16.873064 (+ 38606us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:16.873117 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:16.873199 (+ 82us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:17.352239 (+479040us) raft_consensus.cc:1683] finished
0831 13:58:17.352267 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:17.352433 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5577,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":120,"thread_start_us":38125,"threads_started":1,"wal-append.queue_time_us":132391}
I20250831 13:58:17.327919 21554 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.337428 21503 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.351737 21599 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.373695 21566 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:17.365625 21603 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.390552 21584 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:17.391047 21459 raft_consensus.cc:991] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: : Instructing follower 58aded50ab0d4609a053647eb11b1f48 to start an election
I20250831 13:58:17.391592 21459 raft_consensus.cc:1079] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 LEADER]: Signalling peer 58aded50ab0d4609a053647eb11b1f48 to start an election
I20250831 13:58:17.393730 21582 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.407414 21432 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.408844 21554 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:17.395138 21558 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.395594 21585 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.418094 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:17.421087 20498 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:17.424129 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 817) took 1136 ms. Trace:
I20250831 13:58:17.424551 20498 rpcz_store.cc:276] 0831 13:58:16.285839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.301234 (+ 15395us) service_pool.cc:225] Handling call
0831 13:58:17.417908 (+1116674us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:17.422513 (+ 4605us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3498752}
I20250831 13:58:17.434926 21209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47732 (request call id 55) took 1159 ms. Trace:
I20250831 13:58:17.441520 21612 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:17.437227 20543 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:17.419203 21508 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:17.457197 21209 rpcz_store.cc:276] 0831 13:58:16.275982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.276971 (+ 989us) service_pool.cc:225] Handling call
0831 13:58:16.277371 (+ 400us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:16.277708 (+ 337us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:17.106400 (+828692us) log.cc:844] Serialized 14 byte log entry
0831 13:58:17.434471 (+328071us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.434539 (+ 68us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:17.434597 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:17.434700 (+ 103us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:17.434854 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2129,"thread_start_us":306944,"threads_started":2,"wal-append.queue_time_us":101125}
I20250831 13:58:17.417618 21599 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:17.695662 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 258509us
I20250831 13:58:17.417353 21501 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.701982 21465 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:17.651629 21432 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.697782 21554 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:17.711834 21503 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.697625 21545 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250831 13:58:17.731303 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:17.733479 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 813) took 1448 ms. Trace:
W20250831 13:58:17.734907 21606 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.314s user 0.001s sys 0.000s
I20250831 13:58:17.734025 20501 rpcz_store.cc:276] 0831 13:58:16.284801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.297059 (+ 12258us) service_pool.cc:225] Handling call
0831 13:58:17.731087 (+1434028us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:17.733409 (+ 2322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8824576}
I20250831 13:58:17.717468 21432 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.700340 21612 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:17.724157 21554 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.746876 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 824) took 1601 ms. Trace:
I20250831 13:58:17.747473 20572 rpcz_store.cc:276] 0831 13:58:16.144998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.744340 (+1599342us) service_pool.cc:225] Handling call
0831 13:58:17.744709 (+ 369us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:17.746498 (+ 1789us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:17.746529 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.746572 (+ 43us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:17.746604 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:17.746656 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:17.746808 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:17.748438 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.760389 21595 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:17.767565 20542 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s user 0.028s sys 0.122s
W20250831 13:58:17.767972 20542 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s user 0.028s sys 0.122s
W20250831 13:58:17.773250 21173 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.009s sys 0.033s
W20250831 13:58:17.777550 21173 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.009s sys 0.033s
W20250831 13:58:17.800292 20615 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.007s sys 0.024s
W20250831 13:58:17.804553 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 493586us
I20250831 13:58:17.806054 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:17.801421 20615 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s user 0.007s sys 0.024s
W20250831 13:58:17.813057 20467 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.003s sys 0.016s
W20250831 13:58:17.813512 20467 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.003s sys 0.016s
I20250831 13:58:17.801852 21467 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.803939 21545 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:17.811173 20540 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.032s sys 0.117s
W20250831 13:58:17.807647 21176 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.008s sys 0.016s
I20250831 13:58:17.811038 21479 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:17.855520 20465 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.003s sys 0.021s
W20250831 13:58:17.855923 20465 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.003s sys 0.021s
I20250831 13:58:17.801033 20643 raft_consensus.cc:2953] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:17.835850 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:17.840090 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:17.841203 20650 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.018s sys 0.020s
I20250831 13:58:17.783485 21595 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:17.866935 20650 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s user 0.018s sys 0.020s
I20250831 13:58:17.867585 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.868703 21595 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.868959 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.870723 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.871935 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.872607 20648 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:17.873553 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:17.844035 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 143076us
I20250831 13:58:17.878023 20543 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:17.878791 21595 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 13:58:17.851501 20466 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.009s sys 0.028s
I20250831 13:58:17.849356 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:17.880003 20466 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s user 0.009s sys 0.028s
I20250831 13:58:17.876048 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:17.889024 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:17.888839 20543 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:17.891583 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 51733.1us
I20250831 13:58:17.892655 20543 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:17.815918 21612 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:17.892743 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 56944.2us
I20250831 13:58:17.831139 21503 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:17.839048 20540 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s user 0.033s sys 0.118s
W20250831 13:58:17.848930 21176 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s user 0.008s sys 0.016s
I20250831 13:58:17.897151 20617 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:17.901784 20543 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:17.903688 20568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.018s sys 0.021s
W20250831 13:58:17.904364 20568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s user 0.019s sys 0.022s
W20250831 13:58:17.894167 21551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.008s sys 0.013s
I20250831 13:58:17.905368 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 657) took 1021 ms. Trace:
I20250831 13:58:17.913753 20543 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:58:17.905722 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.027s sys 0.016s
I20250831 13:58:17.917311 20617 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:17.917562 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s user 0.028s sys 0.016s
I20250831 13:58:17.887909 21558 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:17.839463 21501 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20250831 13:58:17.896396 21595 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 13:58:17.923228 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 691929us
I20250831 13:58:17.890949 21584 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:17.894364 21479 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.901652 21432 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:17.902863 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:17.849402 21545 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.907020 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:17.894343 21467 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.915393 20568 rpcz_store.cc:276] 0831 13:58:16.883737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.976834 (+ 93097us) service_pool.cc:225] Handling call
0831 13:58:16.977231 (+ 397us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:16.977512 (+ 281us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:17.171833 (+194321us) log.cc:844] Serialized 14 byte log entry
0831 13:58:17.904962 (+733129us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.905029 (+ 67us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:17.905086 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:17.905168 (+ 82us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:17.905306 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":220441,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":94,"thread_start_us":926638,"threads_started":2}
W20250831 13:58:17.985460 20644 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.023s sys 0.041s
W20250831 13:58:17.915417 21551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.008s sys 0.014s
I20250831 13:58:17.996273 21613 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:17.996953 21613 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:17.939994 20616 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:17.963208 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 49577.6us
W20250831 13:58:17.992377 20644 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s user 0.024s sys 0.041s
I20250831 13:58:17.930585 20397 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59924 (request call id 137) took 3287 ms. Trace:
I20250831 13:58:18.007531 20397 rpcz_store.cc:276] 0831 13:58:14.523520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:14.523685 (+ 165us) service_pool.cc:225] Handling call
0831 13:58:17.810608 (+3286923us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":4}
I20250831 13:58:17.929064 21459 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:17.929220 21464 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:17.956933 21615 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:17.906695 21612 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:17.920579 21554 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:17.959906 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:17.929328 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:18.010545 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 669) took 1118 ms. Trace:
I20250831 13:58:18.057508 20496 raft_consensus.cc:2408] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:18.060104 20644 rpcz_store.cc:276] 0831 13:58:16.892616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.921513 (+ 28897us) service_pool.cc:225] Handling call
0831 13:58:16.957272 (+ 35759us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:16.957462 (+ 190us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:17.313373 (+355911us) log.cc:844] Serialized 14 byte log entry
0831 13:58:18.010142 (+696769us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:18.010208 (+ 66us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:18.010261 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.010352 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.010479 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2595,"prepare.queue_time_us":234326,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":3052,"thread_start_us":1039327,"threads_started":2}
I20250831 13:58:18.063288 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.057258 21625 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.064641 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.019189 20543 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:17.959673 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:18.059199 21206 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
W20250831 13:58:18.069025 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 49965.9us
I20250831 13:58:18.070243 21615 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:17.958758 21617 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:17.929272 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 966653us
I20250831 13:58:18.070807 21615 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:17.958283 21611 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.071201 21617 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.067003 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.072564 21611 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:18.074034 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.074609 20644 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:18.075803 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.076372 20644 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:18.077921 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.069911 20543 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.079599 20543 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.080855 20543 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.081948 20543 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.071777 21615 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:18.067003 21625 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:18.088868 21617 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.089448 21617 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:18.085711 21611 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:18.102334 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 203282us
I20250831 13:58:17.920032 21503 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.010354 21432 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:17.956605 21558 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:17.958432 21614 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:18.021101 21464 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:18.090097 21617 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:18.144896 21463 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.001287 21584 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.105794 21528 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:18.111294 21459 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:18.101357 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.145543 21626 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:18.157231 21545 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.157938 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:18.159250 21479 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.145845 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.160624 21620 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:18.160823 21503 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.112928 21432 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.011663 21508 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20250831 13:58:18.191296 21635 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:18.145538 21463 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.146430 21464 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:18.159422 21558 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:18.270903 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 331034us
W20250831 13:58:18.273046 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 361703us
I20250831 13:58:18.228171 20496 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:18.228328 21626 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:18.278965 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 197090us
I20250831 13:58:18.239953 21467 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:18.272675 21624 log.cc:927] Time spent T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20250831 13:58:18.277464 21528 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:18.275538 20493 raft_consensus.cc:2366] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:18.284109 20541 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:18.272411 21432 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:18.279256 21622 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20250831 13:58:18.274183 21614 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.276039 21620 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.293227 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.294004 21547 raft_consensus.cc:1062] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:18.295781 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 826) took 1134 ms. Trace:
I20250831 13:58:18.299319 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.287974 21626 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:18.299647 21467 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.287585 21558 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.302863 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:18.303644 20616 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.309695 21614 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.306490 21584 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.319062 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 846) took 1144 ms. Trace:
I20250831 13:58:18.325201 20576 rpcz_store.cc:276] 0831 13:58:17.174620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.318243 (+143623us) service_pool.cc:225] Handling call
0831 13:58:17.318516 (+ 273us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:17.319500 (+ 984us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:17.319527 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:17.324079 (+ 4552us) log.cc:844] Serialized 26 byte log entry
0831 13:58:17.765787 (+441708us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:17.907272 (+141485us) log.cc:844] Serialized 14 byte log entry
0831 13:58:17.912872 (+ 5600us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:17.912963 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:18.318792 (+405829us) raft_consensus.cc:1683] finished
0831 13:58:18.318820 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.318984 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5308,"prepare.queue_time_us":379139,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":25957,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":24847,"thread_start_us":564037,"threads_started":2,"wal-append.queue_time_us":628193}
I20250831 13:58:18.307611 21568 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:18.322316 21620 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:18.304898 21528 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.323621 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:18.335693 20577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 808) took 2279 ms (client timeout 2258 ms). Trace:
W20250831 13:58:18.314482 21547 raft_consensus.cc:1066] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Unable to promote non-voter ab7ade51fa134a1b814fab748f01d43d: Service unavailable: leader transfer in progress
I20250831 13:58:18.342271 21648 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:18.337199 20577 rpcz_store.cc:269] 0831 13:58:16.056050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.056188 (+ 138us) service_pool.cc:225] Handling call
0831 13:58:18.333633 (+2277445us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b54000e7754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:18.335607 (+ 1974us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2035538816}
I20250831 13:58:18.324119 21644 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:18.344964 21644 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.345042 21615 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.350740 21644 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:18.341276 21558 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.358639 21528 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:18.355306 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 197713us
I20250831 13:58:18.336833 21568 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:18.394471 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 290400us
W20250831 13:58:18.373790 20617 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.764s (SENT) [suppressed 1 similar messages]
I20250831 13:58:18.334731 21467 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:18.428316 20643 consensus_meta.cc:287] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.535s user 0.012s sys 0.006s
I20250831 13:58:18.383124 21625 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.442787 21615 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.452126 21568 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.459915 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:18.461463 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 663) took 1550 ms. Trace:
W20250831 13:58:18.464500 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 61240.2us
I20250831 13:58:18.461861 20575 rpcz_store.cc:276] 0831 13:58:16.900543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.948832 (+ 48289us) service_pool.cc:225] Handling call
0831 13:58:18.450221 (+1501389us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b5400332a54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:18.450364 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":76455680}
I20250831 13:58:18.443548 21614 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.311245 20494 rpcz_store.cc:276] 0831 13:58:17.161745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.163453 (+ 1708us) service_pool.cc:225] Handling call
0831 13:58:17.168570 (+ 5117us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:17.325477 (+156907us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:17.326895 (+ 1418us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.329269 (+ 2374us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:17.331973 (+ 2704us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.295483 (+963510us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540013d654. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:18.295531 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.295681 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":346186496}
I20250831 13:58:18.474933 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:18.473471 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 135605us
I20250831 13:58:18.475526 20575 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:18.474421 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 828) took 1307 ms. Trace:
I20250831 13:58:18.477667 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.477994 21464 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.479096 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.479792 20644 raft_consensus.cc:2391] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:18.473672 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.480453 20645 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:18.474895 21615 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:18.479696 20576 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 1.
I20250831 13:58:18.482331 21464 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:18.482422 20577 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 3.
I20250831 13:58:18.483584 21432 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.483971 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.484583 20644 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:18.484799 21467 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.485770 21558 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.485410 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.487061 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.487857 20644 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:18.476578 20502 rpcz_store.cc:276] 0831 13:58:17.166930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.205851 (+ 38921us) service_pool.cc:225] Handling call
0831 13:58:18.343069 (+1137218us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:18.343264 (+ 195us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:18.343468 (+ 204us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:18.345053 (+ 1585us) log.cc:844] Serialized 14 byte log entry
0831 13:58:18.474099 (+129046us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:18.474169 (+ 70us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:18.474205 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.474265 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.474358 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36541,"prepare.queue_time_us":1588,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":37296,"spinlock_wait_cycles":11717376,"thread_start_us":92168,"threads_started":1}
I20250831 13:58:18.494268 20467 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:18.497733 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 688) took 1299 ms. Trace:
I20250831 13:58:18.498117 20643 rpcz_store.cc:276] 0831 13:58:17.198467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.259750 (+ 61283us) service_pool.cc:225] Handling call
0831 13:58:17.260241 (+ 491us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:17.289908 (+ 29667us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:17.289932 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:17.300636 (+ 10704us) log.cc:844] Serialized 572 byte log entry
0831 13:58:17.800897 (+500261us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:18.297781 (+496884us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:18.429784 (+132003us) log.cc:844] Serialized 14 byte log entry
0831 13:58:18.476264 (+ 46480us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.476334 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:18.476371 (+ 37us) raft_consensus.cc:1683] finished
0831 13:58:18.497536 (+ 21165us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.497680 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46200,"raft.queue_time_us":46691,"spinlock_wait_cycles":4487040,"thread_start_us":468522,"threads_started":1,"wal-append.queue_time_us":883765}
I20250831 13:58:18.487172 21650 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250831 13:58:18.488296 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.485589 21432 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.482874 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.503129 20466 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:18.527892 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 154143us
I20250831 13:58:18.531529 20467 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:18.515653 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.511240 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.528447 20616 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:18.555408 20497 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 814) took 2260 ms (client timeout 1764 ms). Trace:
I20250831 13:58:18.508395 20496 raft_consensus.cc:1273] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250831 13:58:18.555867 20497 rpcz_store.cc:269] 0831 13:58:16.285079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.302072 (+ 16993us) service_pool.cc:225] Handling call
0831 13:58:18.511108 (+2209036us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:18.545265 (+ 34157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3055872}
I20250831 13:58:18.509964 21595 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.510076 21585 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.542640 21575 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:18.557857 21603 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 13:58:18.588171 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 59839.9us
I20250831 13:58:18.551221 21502 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.596045 20466 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:18.597553 20466 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:18.511749 21554 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.510833 21614 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.509575 21654 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:18.522406 20645 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:18.513561 21565 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.583802 20467 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:18.599812 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.591774 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.608366 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.608381 20466 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:18.579460 21509 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.609920 20467 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:18.637526 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.638312 20575 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
W20250831 13:58:18.665264 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:18.504235 21175 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:18.693554 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.696148 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.582121 21656 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:18.589010 21659 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.590622 21496 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250831 13:58:18.697916 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 88100.2us
W20250831 13:58:18.699637 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 90363.9us
I20250831 13:58:18.593626 21432 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.701077 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.701341 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.591153 21575 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.609588 21581 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:18.707037 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.704324 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:18.527745 21626 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.702718 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.702776 20467 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:18.710209 20643 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:18.710862 20570 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:18.635212 21503 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:18.703269 20575 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:18.707773 21432 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.713266 20570 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:18.714426 21432 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.695490 21508 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.718416 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:18.721118 20467 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:18.698918 21654 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.701859 21558 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:18.716910 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 671) took 1821 ms. Trace:
I20250831 13:58:18.724804 21596 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:18.610644 21554 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.610056 21585 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.717099 21625 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:18.592671 21595 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.699018 21614 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.703737 21620 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.740481 21596 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:18.747342 21620 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:18.748178 21596 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:18.749851 21620 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:18.708663 21575 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:18.727574 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 222239us
I20250831 13:58:18.732287 20543 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.752980 20543 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.753415 20644 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57684
I20250831 13:58:18.709067 21581 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:18.701861 21509 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.639530 21502 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.709793 21648 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.715732 21503 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.726413 21654 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:18.726332 20466 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:18.701444 21659 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.754097 20644 raft_consensus.cc:604] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 LEADER]: Received request to transfer leadership
I20250831 13:58:18.717155 21432 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.737725 21625 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.728919 21558 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.733415 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:18.780478 21651 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:18.781250 21502 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.739214 21603 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:18.768939 20576 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
W20250831 13:58:18.770306 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 44056.3us
I20250831 13:58:18.726012 21582 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:18.771940 20571 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:18.724853 21644 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.727712 20649 rpcz_store.cc:276] 0831 13:58:16.893251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.935047 (+ 41796us) service_pool.cc:225] Handling call
0831 13:58:16.973577 (+ 38530us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:17.325334 (+351757us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:17.326957 (+ 1623us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.329337 (+ 2380us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:17.332016 (+ 2679us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.704133 (+1372117us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b540030edd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:18.707431 (+ 3298us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.714944 (+ 7513us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":794927360}
I20250831 13:58:18.793381 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:18.784844 21591 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 1.580s user 0.007s sys 0.011s
I20250831 13:58:18.768877 21503 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:18.779852 21509 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:18.762871 21581 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:18.784521 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 823) took 2639 ms. Trace:
I20250831 13:58:18.781140 21651 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.794611 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.809984 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.811067 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.812083 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.779161 21663 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:18.779721 21642 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:18.800743 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 821) took 2498 ms. Trace:
I20250831 13:58:18.728400 21508 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.776643 21625 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:18.773527 21467 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:18.782888 21558 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:18.820101 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:18.785135 21603 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.801781 21554 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:18.803615 21502 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:18.801381 21551 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:18.817603 20645 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:18.830278 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.783156 21648 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.833982 21626 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:18.785441 21432 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:18.839563 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.821337 20500 rpcz_store.cc:276] 0831 13:58:16.302377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.840894 (+538517us) service_pool.cc:225] Handling call
0831 13:58:16.841274 (+ 380us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:16.841856 (+ 582us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:16.841895 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:16.842517 (+ 622us) log.cc:844] Serialized 26 byte log entry
0831 13:58:16.992609 (+150092us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:16.992678 (+ 69us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:16.992767 (+ 89us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:18.700087 (+1707320us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:18.796656 (+ 96569us) raft_consensus.cc:1683] finished
0831 13:58:18.796685 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.800670 (+ 3985us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1852,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"spinlock_wait_cycles":8968064,"thread_start_us":149618,"threads_started":1,"wal-append.queue_time_us":268753}
I20250831 13:58:18.848022 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:18.828364 21467 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:18.862489 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.876106 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.828011 21582 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.841887 21603 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:18.805817 20574 rpcz_store.cc:276] 0831 13:58:16.144714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.278068 (+133354us) service_pool.cc:225] Handling call
0831 13:58:17.345147 (+1067079us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b58000f0398. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d757c 00007fb9b99d713b 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:17.345379 (+ 232us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:17.345875 (+ 496us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:17.345919 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:17.346527 (+ 608us) log.cc:844] Serialized 26 byte log entry
0831 13:58:18.493578 (+1147051us) spinlock_profiling.cc:243] Waited 912 us on lock 0x7b1800304f00. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171 00007fb9bc94909f
0831 13:58:18.600492 (+106914us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:18.608415 (+ 7923us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.635742 (+ 27327us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:18.781670 (+145928us) raft_consensus.cc:1683] finished
0831 13:58:18.781698 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.784430 (+ 2732us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3029,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":3291,"spinlock_wait_cycles":7228544,"thread_start_us":93195,"threads_started":1,"wal-append.queue_time_us":1385203}
I20250831 13:58:18.879006 21563 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:18.880167 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.886276 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.853323 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.831140 21642 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:18.892804 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.895416 21626 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:18.863065 21555 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:18.875644 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:18.880700 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.888170 21502 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:18.973589 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:18.947098 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 194184us
I20250831 13:58:18.890965 21467 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:18.997049 20543 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.997367 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.998090 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 430) took 1813 ms. Trace:
I20250831 13:58:18.998670 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.999379 20502 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:19.000450 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:19.001278 20502 raft_consensus.cc:2391] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:19.001374 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:19.002141 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:18.887414 21551 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:19.002460 20543 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.004321 21551 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.004957 21568 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.005541 21568 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.006950 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.007725 20574 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:19.007805 21568 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.003530 20543 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:18.898700 21642 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.019044 20543 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.022205 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.023224 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 818) took 2723 ms. Trace:
I20250831 13:58:19.024698 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 825) took 2879 ms. Trace:
I20250831 13:58:18.974555 21349 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.024629 20495 rpcz_store.cc:276] 0831 13:58:16.286082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.836296 (+550214us) service_pool.cc:225] Handling call
0831 13:58:16.836582 (+ 286us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:16.974875 (+138293us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:16.974907 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:16.974949 (+ 42us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:16.974979 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.994556 (+2019577us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540021c5d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:18.994624 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.009779 (+ 15155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":285417728}
I20250831 13:58:19.027925 20569 rpcz_store.cc:276] 0831 13:58:16.145250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.841694 (+696444us) service_pool.cc:225] Handling call
0831 13:58:16.841961 (+ 267us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:17.052560 (+210599us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:17.052588 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.052648 (+ 60us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:17.052676 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:19.024431 (+1971755us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54003b4f54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:19.024483 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.024625 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":464214400}
I20250831 13:58:19.039142 20543 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.039467 21659 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:18.979468 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:18.978861 21655 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.044062 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.044864 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.022954 21595 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.017216 21502 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:19.051149 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.995501 21555 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.038237 21564 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.043167 21663 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:18.900166 21582 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:19.058178 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:19.024179 21596 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.065335 21210 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47732 (request call id 59) took 1209 ms. Trace:
I20250831 13:58:19.050791 21644 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.071928 20502 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.050992 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.075407 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:18.998484 20646 rpcz_store.cc:276] 0831 13:58:17.184552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.239647 (+ 55095us) service_pool.cc:225] Handling call
0831 13:58:17.239966 (+ 319us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:17.294136 (+ 54170us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:17.294176 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.294224 (+ 48us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:17.294260 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.997853 (+1703593us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b540006a254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:18.997908 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:18.998039 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111039232}
I20250831 13:58:19.044675 21349 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:19.057535 21595 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.067719 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.068145 21564 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:18.975831 21563 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.084046 20497 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.056545 21502 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:19.085568 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.085022 21674 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.082549 20574 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
W20250831 13:58:19.099900 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 218703us
I20250831 13:58:19.100188 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.079172 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.078392 21508 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:19.052597 21655 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.068871 21672 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:19.085186 21432 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.104683 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.107554 20467 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.072381 21176 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.112107 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.117511 20467 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.119053 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.120146 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:19.127259 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:19.127288 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 88222.9us
I20250831 13:58:19.129015 20543 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.131127 21648 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:19.131734 21648 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.132589 21547 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.132346 21648 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.133318 21547 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.135125 21648 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:19.104058 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.136093 20647 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:19.136488 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.136992 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:19.137567 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.137794 21349 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:19.138068 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.075064 21210 rpcz_store.cc:276] 0831 13:58:17.914811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.931073 (+ 16262us) service_pool.cc:225] Handling call
0831 13:58:17.957682 (+ 26609us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:18.110840 (+153158us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:18.115667 (+ 4827us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:18.132823 (+ 17156us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:18.144325 (+ 11502us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:19.064931 (+920606us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54004001d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:19.064996 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.065157 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":311108480}
I20250831 13:58:19.115907 20645 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:19.137571 20647 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.139261 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.139472 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.139917 20569 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:19.140704 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.085553 21503 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.115332 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.145910 20466 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.140880 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.138665 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:19.190551 20574 raft_consensus.cc:2391] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:19.191124 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 45328.9us
I20250831 13:58:19.196861 21613 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:19.197535 21613 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.144627 21584 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.150735 21663 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:19.152246 21568 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.114382 21651 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:19.163480 21626 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.146212 21547 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:19.227089 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.186182 21432 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:19.141218 20385 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 261368us
I20250831 13:58:19.217677 21349 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:19.194648 21556 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.248399 21349 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:19.186378 20645 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:19.250911 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.251806 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 842) took 2078 ms. Trace:
I20250831 13:58:19.252292 20573 rpcz_store.cc:276] 0831 13:58:17.173454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.279985 (+106531us) service_pool.cc:225] Handling call
0831 13:58:18.450479 (+1170494us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:18.457557 (+ 7078us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:18.457599 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:18.470016 (+ 12417us) log.cc:844] Serialized 26 byte log entry
0831 13:58:18.695342 (+225326us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:18.695383 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.695451 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:18.836081 (+140630us) raft_consensus.cc:1683] finished
0831 13:58:18.836105 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.251516 (+415411us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400332a50. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:19.251724 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4050,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":3751,"spinlock_wait_cycles":2655451008,"thread_start_us":210794,"threads_started":1,"wal-append.queue_time_us":294838}
I20250831 13:58:19.256088 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.257505 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.199998 20493 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:19.264546 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.221632 21584 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.215581 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.270713 21643 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:19.270789 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.239319 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.245290 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:19.272245 21210 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:19.272591 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:19.251837 20648 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 692) took 1368 ms (client timeout 1547 ms). Trace:
I20250831 13:58:19.272282 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:19.279987 20648 rpcz_store.cc:269] 0831 13:58:17.883564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.900913 (+ 17349us) service_pool.cc:225] Handling call
0831 13:58:19.250741 (+1349828us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:19.251746 (+ 1005us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7000320}
I20250831 13:58:19.230752 20569 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:19.263566 20616 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:19.221395 21672 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.169624 21551 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.265799 21604 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.268638 21596 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.270874 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.254567 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.281042 20571 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
W20250831 13:58:19.284734 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 157531us
I20250831 13:58:19.302551 20467 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.304021 20467 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.305526 20467 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:19.288208 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 159303us
I20250831 13:58:19.300334 20502 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:19.306792 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.278345 21584 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.306419 21549 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:19.269760 21432 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.309962 21551 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 13:58:19.310642 21670 log.cc:927] Time spent T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.516s user 0.000s sys 0.003s
I20250831 13:58:19.310752 21503 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.311360 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:19.312489 21642 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:19.329932 21642 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.333384 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.314826 20467 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.317866 21655 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.304723 21643 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:19.318733 20647 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:19.335867 21643 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:19.321010 21582 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.336818 20644 raft_consensus.cc:1238] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.337400 20568 raft_consensus.cc:1238] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.320959 20646 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57684
I20250831 13:58:19.321621 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 832) took 1398 ms. Trace:
I20250831 13:58:19.338312 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:19.321955 21432 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.322906 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:19.307197 21604 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.314554 21349 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:19.341012 20646 raft_consensus.cc:604] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 LEADER]: Received request to transfer leadership
I20250831 13:58:19.337913 20499 rpcz_store.cc:276] 0831 13:58:17.923264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.923430 (+ 166us) service_pool.cc:225] Handling call
0831 13:58:17.951552 (+ 28122us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:18.227785 (+276233us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:18.227815 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:18.525543 (+297728us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b2c00035a28. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bb3199a1 00007fb9bb328fd2 00007fb9bc9b2021 00007fb9bc9763fa 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32
0831 13:58:18.526029 (+ 486us) log.cc:844] Serialized 26 byte log entry
0831 13:58:18.581296 (+ 55267us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:18.581352 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:18.581457 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:19.259265 (+677808us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:19.321390 (+ 62125us) raft_consensus.cc:1683] finished
0831 13:58:19.321410 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.321540 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":192694,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":49087,"spinlock_wait_cycles":57517696,"thread_start_us":330648,"threads_started":2,"wal-append.queue_time_us":177196}
I20250831 13:58:19.344141 21210 raft_consensus.cc:1811] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Handling vote request from an unknown peer 44d163094add49049cc749547d891602
I20250831 13:58:19.341933 21582 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 13:58:19.350634 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 87147.9us
I20250831 13:58:19.354552 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:19.347064 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.359691 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.341761 21646 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:19.319896 21563 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.348228 21655 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:19.364620 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.347007 21604 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.380471 20617 leader_election.cc:400] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:19.366232 20616 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:19.381830 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.387216 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:19.387416 21661 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.387882 21661 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:19.388218 21661 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.388945 21479 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:19.389321 21479 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.389534 21568 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.389686 21479 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:19.390071 21568 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.388248 20467 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.392004 21568 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.392288 21562 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.392647 21568 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.392822 21562 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.393263 21562 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.393206 21568 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.386234 20617 leader_election.cc:403] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250831 13:58:19.354970 21555 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:19.400468 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 93774.2us
I20250831 13:58:19.401273 20543 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.362936 21349 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.416847 20543 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:19.450940 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 84461.8us
W20250831 13:58:19.451532 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 54633.7us
W20250831 13:58:19.469398 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 72685.7us
I20250831 13:58:19.452487 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.470172 20468 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:19.381407 21596 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.463639 21210 raft_consensus.cc:2408] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20250831 13:58:19.473881 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 119524us
I20250831 13:58:19.384821 21459 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:19.395469 21568 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.385782 21655 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:19.531193 21655 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:19.535950 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.498602 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:19.536567 21654 consensus_meta.cc:287] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.760s user 0.007s sys 0.011s
I20250831 13:58:19.472738 21431 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.494755 21559 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
W20250831 13:58:19.538388 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 62438.1us
W20250831 13:58:19.538969 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 122211us
I20250831 13:58:19.536384 21609 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:19.498445 20541 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.539292 20617 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.540779 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.397301 21563 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.472852 21672 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:19.548331 21568 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.468490 21349 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:19.472440 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 410905us
I20250831 13:58:19.500664 21459 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:19.550050 21581 consensus_meta.cc:287] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.732s user 0.012s sys 0.009s
I20250831 13:58:19.568627 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:19.570113 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 176042us
I20250831 13:58:19.585850 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.585871 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:19.559379 21603 consensus_meta.cc:287] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.662s user 0.010s sys 0.000s
W20250831 13:58:19.590704 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 94830.7us
I20250831 13:58:19.596951 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.615607 21603 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.616146 20541 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:19.616500 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:19.617174 20541 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:19.617563 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.618845 21609 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.619228 21609 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.619573 21609 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.620029 21431 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:19.620652 21431 raft_consensus.cc:695] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:19.623332 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.548673 21547 consensus_queue.cc:1046] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250831 13:58:19.624059 20499 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:19.624475 21547 raft_consensus.cc:3053] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:19.624843 21547 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:19.625020 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.625978 20499 raft_consensus.cc:2391] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:19.625638 21547 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.593568 21493 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.586695 21568 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:19.590286 21349 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.628516 21562 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.597991 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.632846 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.633886 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 687) took 2435 ms. Trace:
I20250831 13:58:19.634236 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:19.558058 21672 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:19.634905 21672 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:19.635298 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.634210 20650 rpcz_store.cc:276] 0831 13:58:17.198172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.275284 (+ 77112us) service_pool.cc:225] Handling call
0831 13:58:17.285011 (+ 9727us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:17.312574 (+ 27563us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:17.312840 (+ 266us) log.cc:844] Serialized 14 byte log entry
0831 13:58:17.896778 (+583938us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:17.896835 (+ 57us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:17.896875 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:19.633666 (+1736791us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b54002eb154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:19.633704 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.633816 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30215,"prepare.queue_time_us":803,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":32030,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":5411,"spinlock_wait_cycles":62873344,"thread_start_us":554215,"threads_started":1,"wal-append.queue_time_us":686342,"wal-append.run_cpu_time_us":9453,"wal-append.run_wall_time_us":1149481}
I20250831 13:58:19.634250 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 435) took 1800 ms. Trace:
I20250831 13:58:19.637145 20651 rpcz_store.cc:276] 0831 13:58:17.833383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.836481 (+ 3098us) service_pool.cc:225] Handling call
0831 13:58:19.633580 (+1797099us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d75ce 000000000035a3c9 00007fb9c185e300 00007fb9c1924ec4 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:19.633733 (+ 153us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:19.633925 (+ 192us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:19.633953 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:19.634002 (+ 49us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:19.634034 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:19.634093 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.634195 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24405376}
I20250831 13:58:19.640223 21479 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.603922 21596 consensus_queue.cc:1046] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 13:58:19.604933 21683 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250831 13:58:19.551437 21654 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.608662 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:19.610100 21467 consensus_meta.cc:287] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.583s user 0.013s sys 0.005s
I20250831 13:58:19.627100 21568 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:19.603261 21684 log.cc:927] Time spent T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20250831 13:58:19.644160 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.645360 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.646217 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.608024 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:19.658955 20499 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:19.659062 20570 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.649761 20568 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:19.617563 21603 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.661670 20466 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.556306 21661 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.664337 20466 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:19.649449 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:19.593523 21630 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:19.651903 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 449) took 1069 ms. Trace:
I20250831 13:58:19.665354 20643 rpcz_store.cc:276] 0831 13:58:18.583005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:18.591474 (+ 8469us) service_pool.cc:225] Handling call
0831 13:58:19.651757 (+1060283us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b540015bdd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:19.651854 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177325824}
I20250831 13:58:19.651980 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.596118 21459 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:19.595913 21542 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 13:58:19.654505 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 184418us
W20250831 13:58:19.654814 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 115654us
I20250831 13:58:19.602367 21655 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.661180 20467 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.621822 21431 consensus_queue.cc:237] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.667953 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.668965 21659 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:19.671577 20574 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.647766 21547 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.646009 21687 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.676669 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.630345 21643 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:19.681871 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.643167 21581 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.672626 21596 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.672593 21542 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.689981 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.691689 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 866) took 1057 ms. Trace:
I20250831 13:58:19.693435 21479 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.709136 21542 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.710276 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:19.681624 21625 consensus_meta.cc:287] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.844s user 0.011s sys 0.006s
W20250831 13:58:19.699179 21502 consensus_meta.cc:287] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.587s user 0.001s sys 0.008s
I20250831 13:58:19.688149 21659 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.705024 20572 rpcz_store.cc:276] 0831 13:58:18.634704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:18.717221 (+ 82517us) service_pool.cc:225] Handling call
0831 13:58:18.717514 (+ 293us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:19.042027 (+324513us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:19.042071 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:19.051931 (+ 9860us) log.cc:844] Serialized 26 byte log entry
0831 13:58:19.232005 (+180074us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:19.232040 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:19.232098 (+ 58us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:19.691504 (+459406us) raft_consensus.cc:1683] finished
0831 13:58:19.691520 (+ 16us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.691639 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":103373,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":19151,"thread_start_us":439954,"threads_started":2,"wal-append.queue_time_us":298666}
I20250831 13:58:19.721027 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:19.706543 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 83655.6us
I20250831 13:58:19.708204 20574 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:19.707389 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.694458 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.722353 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.683564 21603 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:19.722549 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.723309 21603 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.659952 21467 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.723704 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.724149 20497 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.723721 21603 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:19.724126 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:19.725044 21467 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:19.699182 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 137273us
I20250831 13:58:19.701180 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.725720 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.726137 20497 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac"
dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
from {username='slave'} at 127.0.0.1:52988
W20250831 13:58:19.726377 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:19.722978 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:19.727185 20573 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:19.727335 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.728272 20574 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:19.729490 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.733096 21674 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.732751 21603 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:19.734005 20497 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:58:19.733711 21674 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.734408 20497 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:19.734910 20572 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:19.735750 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.736016 21674 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.736718 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.737812 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.737381 20543 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:19.740579 20497 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:19.740849 21674 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:19.742439 21674 raft_consensus.cc:695] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:19.704836 21562 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.743427 21674 consensus_queue.cc:237] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.680642 21692 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:19.737721 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:19.747277 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 48160us
I20250831 13:58:19.747468 20574 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:19.749716 20573 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:19.750643 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:19.752597 20568 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
W20250831 13:58:19.767884 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 83435us
I20250831 13:58:19.705710 21687 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:19.773361 21687 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.774039 21693 consensus_queue.cc:1046] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20250831 13:58:19.759930 20496 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:19.780061 20499 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.749094 21659 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:19.778970 20543 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:19.779266 20616 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:19.787904 20572 raft_consensus.cc:1273] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:19.786314 20543 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:19.782191 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 92222us
I20250831 13:58:19.793171 20570 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:19.794570 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:19.795388 20570 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:19.797078 20543 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.798585 20543 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.801009 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 833) took 1877 ms. Trace:
I20250831 13:58:19.718258 21625 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:19.801533 20501 rpcz_store.cc:276] 0831 13:58:17.923636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.931260 (+ 7624us) service_pool.cc:225] Handling call
0831 13:58:17.956354 (+ 25094us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:18.199485 (+243131us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:18.199519 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:18.199560 (+ 41us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:18.199594 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:19.800711 (+1601117us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54001284d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:19.800786 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:19.800942 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":558669056}
I20250831 13:58:19.804104 20543 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.746856 21502 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.809985 20501 raft_consensus.cc:1273] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:19.811117 20501 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:19.807030 20616 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:19.809316 21681 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.825412 21679 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:19.821918 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 43776.8us
I20250831 13:58:19.802727 21552 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.802196 21595 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:19.838464 20569 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:19.839907 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:19.840781 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:19.822105 21575 consensus_meta.cc:287] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.071s user 0.021s sys 0.004s
I20250831 13:58:19.818275 21603 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.784097 21584 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.779690 21604 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.818910 21654 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.835265 21681 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.860124 20568 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.855733 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.851954 21349 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.845882 21208 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:19.861510 21681 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:19.869184 21691 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20250831 13:58:19.869421 20572 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.851920 21583 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.176s
I20250831 13:58:19.855357 21661 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:19.894009 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 145924us
W20250831 13:58:19.902181 20577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34436 (request call id 13) took 1429 ms (client timeout 1535 ms). Trace:
I20250831 13:58:19.862612 21655 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:19.902642 20577 rpcz_store.cc:269] 0831 13:58:18.473687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:18.474319 (+ 632us) service_pool.cc:225] Handling call
0831 13:58:19.902013 (+1427694us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b540031e2d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:19.902125 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9473408}
I20250831 13:58:19.904376 20617 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:19.908547 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 101583us
I20250831 13:58:19.847829 21575 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.848500 21555 consensus_queue.cc:1046] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
W20250831 13:58:19.907676 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 112832us
I20250831 13:58:19.911001 20570 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:19.880498 21595 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:19.910238 20617 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.890754 21431 consensus_queue.cc:1046] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20250831 13:58:19.917405 21697 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.918725 20568 raft_consensus.cc:1238] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:19.892689 21583 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:19.908303 21581 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.928154 20617 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.882516 21680 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.925302 20467 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.938843 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:19.918788 21661 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:19.927841 21643 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.891829 21176 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:19.935078 21583 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.935174 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:19.939319 21697 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:19.975914 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.934558 21595 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:19.959968 21655 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:19.965951 20499 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.976797 20467 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.959755 21625 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.980496 21584 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:19.945933 21680 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:19.994462 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:19.967640 21583 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:20.009398 20467 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:19.968557 21687 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:19.964666 21661 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:20.010885 20467 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:19.983455 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 65935.1us
I20250831 13:58:20.013669 20573 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:19.978613 21697 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.025802 21588 raft_consensus.cc:3053] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:20.026304 21588 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:20.031867 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 676) took 1300 ms. Trace:
I20250831 13:58:20.032709 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 439) took 1187 ms. Trace:
I20250831 13:58:20.033517 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.034335 20498 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
W20250831 13:58:20.034955 20498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 426) took 2200 ms (client timeout 1717 ms). Trace:
I20250831 13:58:19.992215 21643 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:20.035279 20498 rpcz_store.cc:269] 0831 13:58:17.834838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.880755 (+ 45917us) service_pool.cc:225] Handling call
0831 13:58:20.033312 (+2152557us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:20.034877 (+ 1565us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23860480}
I20250831 13:58:20.032418 20569 rpcz_store.cc:276] 0831 13:58:18.731293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.281950 (+550657us) service_pool.cc:225] Handling call
0831 13:58:19.306741 (+ 24791us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:19.839274 (+532533us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.031554 (+192280us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54002a8fd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:20.031777 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1222249728}
I20250831 13:58:20.030061 21702 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
W20250831 13:58:20.038934 21704 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20250831 13:58:19.981534 21655 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:20.033182 20495 rpcz_store.cc:276] 0831 13:58:18.845449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.069891 (+224442us) service_pool.cc:225] Handling call
0831 13:58:20.031833 (+961942us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:20.032118 (+ 285us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.032315 (+ 197us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:20.032352 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:20.032406 (+ 54us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:20.032448 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.032519 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:20.032647 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6457600}
I20250831 13:58:20.039304 21707 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.106s
I20250831 13:58:20.005326 21502 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.091926 20498 raft_consensus.cc:1238] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.047998 21643 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:20.095422 21643 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:20.022706 21585 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:19.981333 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 89604.7us
I20250831 13:58:20.056548 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.006580 21680 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.036360 21584 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 13:58:20.099021 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 47018.6us
I20250831 13:58:20.100070 20499 raft_consensus.cc:1238] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 5. Ops: []
I20250831 13:58:20.111297 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:20.042755 21604 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:20.035445 20642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 433) took 2185 ms (client timeout 1988 ms). Trace:
I20250831 13:58:20.059445 21655 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.138152 20467 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:20.060386 21349 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:20.098548 20494 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.152745 20498 raft_consensus.cc:1238] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:20.060175 21661 consensus_queue.cc:1046] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20250831 13:58:20.168124 20499 raft_consensus.cc:1238] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.140213 21584 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:20.046829 21696 consensus_queue.cc:1046] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.106s
I20250831 13:58:20.153129 21575 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:20.180750 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44497.6us
I20250831 13:58:20.175010 21497 consensus_queue.cc:237] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:20.185230 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.185410 21712 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:20.186091 21207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47742 (request call id 42) took 2201 ms. Trace:
W20250831 13:58:20.162578 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 142148us
I20250831 13:58:20.175863 21584 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.158012 21568 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:20.172547 21175 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.188985 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 835) took 2256 ms. Trace:
I20250831 13:58:20.205209 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47742 (request call id 45) took 1371 ms. Trace:
I20250831 13:58:20.209324 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.194211 21692 raft_consensus.cc:2762] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20250831 13:58:20.197137 21712 raft_consensus.cc:695] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:20.217561 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 289433us
I20250831 13:58:20.218114 20502 raft_consensus.cc:1811] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Handling vote request from an unknown peer 44d163094add49049cc749547d891602
W20250831 13:58:20.144582 20642 rpcz_store.cc:269] 0831 13:58:17.830232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.836937 (+ 6705us) service_pool.cc:225] Handling call
0831 13:58:19.926398 (+2089461us) spinlock_profiling.cc:243] Waited 5.13 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:20.012117 (+ 85719us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11804416}
I20250831 13:58:20.219898 20502 raft_consensus.cc:2391] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:20.220322 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.206117 21580 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.196578 21604 consensus_queue.cc:1046] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250831 13:58:20.226194 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.225522 20617 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.216989 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 445) took 1218 ms. Trace:
W20250831 13:58:20.221663 21558 consensus_meta.cc:287] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.373s user 0.010s sys 0.010s
I20250831 13:58:20.214937 21595 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.309023 21673 consensus_queue.cc:1046] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
W20250831 13:58:20.313777 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 88326.6us
W20250831 13:58:20.232529 21672 consensus_meta.cc:287] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.597s user 0.008s sys 0.002s
I20250831 13:58:20.329152 20617 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.218057 21687 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:20.223656 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 66953.6us
I20250831 13:58:20.321689 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.229166 20493 rpcz_store.cc:276] 0831 13:58:18.998828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.038622 (+ 39794us) service_pool.cc:225] Handling call
0831 13:58:20.216779 (+1178157us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54001c2854. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:20.216924 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":231712512}
I20250831 13:58:20.232033 21463 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:20.318933 21349 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.334115 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.384815 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:20.386915 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.180945 21588 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.388055 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.389017 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.389976 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.391685 21568 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:20.329106 21661 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.329285 21552 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:20.211642 20494 rpcz_store.cc:276] 0831 13:58:17.924234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:18.477273 (+553039us) service_pool.cc:225] Handling call
0831 13:58:20.012511 (+1535238us) spinlock_profiling.cc:243] Waited 4.55 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:20.045096 (+ 32585us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.161330 (+116234us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.181078 (+ 19748us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10466560}
W20250831 13:58:20.384209 21609 consensus_meta.cc:287] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.764s user 0.003s sys 0.007s
I20250831 13:58:20.401369 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.409822 20499 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:20.213436 21208 rpcz_store.cc:276] 0831 13:58:18.833921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:18.850994 (+ 17073us) service_pool.cc:225] Handling call
0831 13:58:18.862139 (+ 11145us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:19.868636 (+1006497us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.193762 (+325126us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b54005e1ad4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:20.205135 (+ 11373us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2235638144}
I20250831 13:58:20.405886 21349 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:20.416210 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:20.419103 21714 log.cc:927] Time spent T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
I20250831 13:58:20.419252 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.420868 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.423053 20543 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.328454 21558 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.422030 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.399248 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.334460 21672 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.430328 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.445670 21561 raft_consensus.cc:3053] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:20.446039 21561 raft_consensus.cc:738] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:20.446534 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.446795 20543 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.447841 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.448055 21630 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:20.448458 21630 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:20.408505 21552 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:20.424907 21459 consensus_meta.cc:287] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.743s user 0.006s sys 0.005s
I20250831 13:58:20.186581 21207 rpcz_store.cc:276] 0831 13:58:17.958704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:17.958954 (+ 250us) service_pool.cc:225] Handling call
0831 13:58:17.959219 (+ 265us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:18.112575 (+153356us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 13:58:18.289151 (+176576us) log.cc:844] Serialized 14 byte log entry
0831 13:58:18.433445 (+144294us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:18.435395 (+ 1950us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:18.435453 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.138899 (+1703446us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54005e0e54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:20.145468 (+ 6569us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:20.157960 (+ 12492us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":68439,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":1823,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":2038,"spinlock_wait_cycles":305905408,"thread_start_us":182467,"threads_started":2,"wal-append.queue_time_us":176327,"wal-append.run_cpu_time_us":1217,"wal-append.run_wall_time_us":1071882}
I20250831 13:58:20.441237 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.442065 21596 raft_consensus.cc:3053] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:20.451460 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.422029 21491 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:20.457104 20643 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:20.460690 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.458264 20500 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:20.461550 20500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 431) took 1621 ms (client timeout 1522 ms). Trace:
W20250831 13:58:20.461895 20500 rpcz_store.cc:269] 0831 13:58:18.840046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:18.876468 (+ 36422us) service_pool.cc:225] Handling call
0831 13:58:20.441071 (+1564603us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:20.461487 (+ 20416us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10058496}
I20250831 13:58:20.426340 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:20.426719 21674 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.464218 20572 raft_consensus.cc:2366] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4.
I20250831 13:58:20.457556 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:20.450950 21542 consensus_meta.cc:287] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.741s user 0.008s sys 0.003s
I20250831 13:58:20.468421 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.415159 21609 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.471652 20574 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.472601 21609 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.475206 20500 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:20.476157 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:20.455711 21473 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.459512 21596 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:20.459929 21491 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:20.487885 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.486260 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:20.490868 20577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.055s sys 0.096s
W20250831 13:58:20.491703 20577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s user 0.055s sys 0.096s
I20250831 13:58:20.492791 20500 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:20.489238 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.494573 20643 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:20.496153 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.494482 20543 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:20.497859 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:20.491250 20570 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.023s sys 0.060s
I20250831 13:58:20.481020 21720 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:20.481731 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:20.458556 21459 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.505261 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:20.498852 20570 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.523s user 0.024s sys 0.060s
I20250831 13:58:20.506942 20466 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.509554 20466 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.489606 21491 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:20.529479 20541 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.531613 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.535038 20643 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:20.478044 21542 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.537838 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.483707 21609 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250831 13:58:20.540462 20646 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.537588 20468 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:20.431334 21712 consensus_queue.cc:237] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.467221 21561 consensus_queue.cc:260] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.547999 21561 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:20.538535 21659 consensus_meta.cc:287] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.720s user 0.007s sys 0.003s
I20250831 13:58:20.500649 21702 raft_consensus.cc:695] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:20.541195 21412 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.534226 21642 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:20.535885 21568 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:20.529661 21552 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.537999 21473 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:20.574805 20569 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.617688 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:20.603123 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:20.598275 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:20.552625 21676 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:20.554106 21673 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:20.553460 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.626794 20500 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:20.553742 21693 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250831 13:58:20.626374 21562 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.009s sys 0.010s
I20250831 13:58:20.507290 21720 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.579185 21473 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:20.678817 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 720) took 1062 ms. Trace:
I20250831 13:58:20.680325 20574 rpcz_store.cc:276] 0831 13:58:19.616193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.748204 (+132011us) service_pool.cc:225] Handling call
0831 13:58:19.778602 (+ 30398us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.472006 (+693404us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.678558 (+206552us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b54003b4a54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:20.678753 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1593450240}
W20250831 13:58:20.662839 21583 consensus_meta.cc:287] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.644s user 0.012s sys 0.001s
W20250831 13:58:20.673480 21684 log.cc:927] Time spent T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
W20250831 13:58:20.665477 21563 consensus_meta.cc:287] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.095s user 0.012s sys 0.000s
I20250831 13:58:20.692517 20643 raft_consensus.cc:2408] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:20.664203 21558 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.574486 21659 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.746358 20570 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.749833 20497 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.749030 20466 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:20.680061 21562 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.010s sys 0.011s
I20250831 13:58:20.578940 21635 consensus_queue.cc:1046] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.109s
I20250831 13:58:20.672621 21672 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:20.748214 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 424103us
I20250831 13:58:20.755239 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:52988 (request call id 845) took 1438 ms. Trace:
I20250831 13:58:20.691089 21473 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.755734 20497 rpcz_store.cc:276] 0831 13:58:19.312273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.726010 (+413737us) service_pool.cc:225] Handling call
0831 13:58:20.693804 (+967794us) spinlock_profiling.cc:243] Waited 7.3 ms on lock 0x7b54000113d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc96d9fd 00007fb9c18d5880 00007fb9baf6dc44 00007fb9baf6dbd2 00007fb9baf6db01 00007fb9baf6da7c 00007fb9baf6cd32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:20.750205 (+ 56401us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16787328}
I20250831 13:58:20.767012 20497 raft_consensus.cc:1238] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.766887 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.767556 20643 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:20.768194 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 85221.2us
I20250831 13:58:20.670739 21676 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:20.574637 21596 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:20.777096 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 447977us
I20250831 13:58:20.779778 20617 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.753168 21563 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.782064 20617 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.784619 20466 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.755194 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.784863 20617 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.787966 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.788442 20574 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:20.792227 20617 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.778353 21676 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.781469 21580 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.808395 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:20.807816 20576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 850) took 2208 ms (client timeout 1800 ms). Trace:
W20250831 13:58:20.815618 20576 rpcz_store.cc:269] 0831 13:58:18.599422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:18.599571 (+ 149us) service_pool.cc:225] Handling call
0831 13:58:20.795753 (+2196182us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b54000e7254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:20.807751 (+ 11998us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":314608000}
I20250831 13:58:20.811833 20643 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:20.781841 21581 consensus_queue.cc:1046] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250831 13:58:20.777279 21705 consensus_queue.cc:1046] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20250831 13:58:20.816183 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.827672 21549 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:20.828832 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.794979 21725 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:20.829046 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.795344 21651 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.830022 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.750574 21702 consensus_queue.cc:237] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.830570 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:20.831727 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:20.833274 21730 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.834040 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.835714 20466 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.785969 21412 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:20.793078 21596 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.834522 21720 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:20.837209 20466 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:20.845697 21725 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.836192 20541 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.838467 20616 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:20.846661 21726 log.cc:927] Time spent T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250831 13:58:20.847038 21552 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.817097 20570 raft_consensus.cc:1238] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.782781 21672 consensus_queue.cc:1046] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
W20250831 13:58:20.841135 21681 consensus_meta.cc:287] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.972s user 0.004s sys 0.008s
I20250831 13:58:20.841897 21651 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 13:58:20.845347 21680 consensus_meta.cc:287] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.699s user 0.007s sys 0.004s
I20250831 13:58:20.808303 21661 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.849624 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:20.843278 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 467) took 1192 ms. Trace:
I20250831 13:58:20.859333 20647 rpcz_store.cc:276] 0831 13:58:19.644083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.649089 (+ 5006us) service_pool.cc:225] Handling call
0831 13:58:20.751690 (+1102601us) spinlock_profiling.cc:243] Waited 3.92 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d766c 00007fb9b99d75d6 000000000035a3c9 00007fb9c185e300 00007fb9c1924ec4 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:20.836835 (+ 85145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9025920}
I20250831 13:58:20.861004 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.711539 21583 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.858178 20500 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:20.862327 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.867569 21705 consensus_queue.cc:1230] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Peer 212b66976f874aa682cae0fb77928786 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 13:58:20.837177 21459 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:20.854955 21721 log.cc:927] Time spent T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20250831 13:58:20.868404 20541 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.870196 21727 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.852488 21412 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.863556 21680 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.871448 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.872157 21412 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.872738 21412 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.872798 21680 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.873342 21210 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 3 LEARNER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:20.872156 20647 raft_consensus.cc:3053] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:20.874467 20647 raft_consensus.cc:738] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:20.875290 20647 consensus_queue.cc:260] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
W20250831 13:58:20.876389 21655 consensus_meta.cc:287] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.718s user 0.007s sys 0.001s
I20250831 13:58:20.816826 20494 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.876617 20647 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 5
I20250831 13:58:20.876811 21655 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.877825 20495 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:20.863863 21580 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:20.880180 21547 consensus_meta.cc:287] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 1.169s user 0.011s sys 0.000s
W20250831 13:58:20.883061 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 45965.4us
I20250831 13:58:20.884054 20466 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.884941 20647 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:20.855772 21687 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:20.886807 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.887835 20466 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.888216 20647 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:20.891310 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:20.892190 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:20.892560 20497 raft_consensus.cc:1238] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:20.893281 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.893611 20466 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:20.893968 20500 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:20.877874 21674 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.901386 20647 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:20.866669 21725 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:20.922135 20541 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:20.922705 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:20.925968 21459 consensus_queue.cc:1046] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:20.927415 21459 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:20.927503 21547 consensus_queue.cc:1046] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:20.868106 21720 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:20.929373 21547 raft_consensus.cc:3053] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:20.929404 21720 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:20.930053 21547 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:20.930226 21720 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:20.870205 21174 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.893734 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.856681 21681 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.881417 21630 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.887164 21562 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:20.933411 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.938488 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 726) took 1280 ms. Trace:
I20250831 13:58:20.896657 21672 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:20.893563 21596 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:20.940368 20647 rpcz_store.cc:276] 0831 13:58:19.658278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.933210 (+1274932us) service_pool.cc:225] Handling call
0831 13:58:20.938445 (+ 5235us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:20.894021 21642 raft_consensus.cc:3053] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:20.932850 21501 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:20.895183 21687 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:20.947971 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:20.946200 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:20.923259 21558 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:20.887069 21412 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20250831 13:58:20.955173 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.940009 21659 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:20.949353 21642 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:20.954624 21501 raft_consensus.cc:695] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:20.920270 21722 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:20.971352 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:20.965202 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 709) took 2159 ms. Trace:
I20250831 13:58:20.973417 20644 rpcz_store.cc:276] 0831 13:58:18.805502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.265324 (+459822us) service_pool.cc:225] Handling call
0831 13:58:19.336649 (+ 71325us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:19.337182 (+ 533us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.964970 (+1627788us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b540028a850. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:20.965131 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":74932096}
I20250831 13:58:20.977370 20574 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:20.978591 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:20.931278 21676 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.984280 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:20.952965 21727 consensus_queue.cc:1046] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20250831 13:58:20.964187 21642 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.968986 21635 consensus_queue.cc:1046] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20250831 13:58:20.958593 21687 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:20.987285 20400 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59926 (request call id 132) took 6561 ms. Trace:
I20250831 13:58:20.987358 21634 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:20.997430 20400 rpcz_store.cc:276] 0831 13:58:14.425574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:14.425713 (+ 139us) service_pool.cc:225] Handling call
0831 13:58:20.987094 (+6561381us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":128139,"reported_tablets":2}
I20250831 13:58:20.989164 20650 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:21.003883 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 468) took 1353 ms. Trace:
I20250831 13:58:20.940176 21459 consensus_queue.cc:1046] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250831 13:58:20.991722 20466 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:20.955381 21672 raft_consensus.cc:695] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:20.936447 21725 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.952189 21562 raft_consensus.cc:695] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:20.953292 21680 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:21.001855 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:21.004532 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:20.990037 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:21.014261 20647 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:21.014844 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:21.004252 20650 rpcz_store.cc:276] 0831 13:58:19.650652 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.650807 (+ 155us) service_pool.cc:225] Handling call
0831 13:58:20.988868 (+1338061us) spinlock_profiling.cc:243] Waited 6.99 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d75ce 000000000035a3c9 00007fb9c185e300 00007fb9c1924ec4 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:20.989004 (+ 136us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.003664 (+ 14660us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.003834 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16423040}
I20250831 13:58:21.003818 20500 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.016111 20466 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:21.016893 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:21.017839 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:21.007656 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 85616.7us
I20250831 13:58:20.969563 21722 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.995812 21659 raft_consensus.cc:3053] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:21.019340 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:21.019271 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:21.059703 21701 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:21.023533 21583 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:21.023962 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 733) took 1353 ms. Trace:
I20250831 13:58:21.025916 21681 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.029575 20647 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:21.029999 21585 raft_consensus.cc:3053] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:21.060230 21701 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.060499 20650 rpcz_store.cc:276] 0831 13:58:19.670997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.016208 (+1345211us) service_pool.cc:225] Handling call
0831 13:58:21.023476 (+ 7268us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.023641 (+ 165us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.023668 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:21.023712 (+ 44us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:21.023744 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.023798 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.023904 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.061034 21585 raft_consensus.cc:738] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:21.061944 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 469) took 1305 ms. Trace:
W20250831 13:58:21.030294 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 171063us
W20250831 13:58:21.062294 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:21.062345 20647 rpcz_store.cc:276] 0831 13:58:19.756493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.061212 (+1304719us) service_pool.cc:225] Handling call
0831 13:58:21.061464 (+ 252us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.061642 (+ 178us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.061669 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:21.061709 (+ 40us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:21.061738 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.061790 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.061889 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.009995 21642 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:21.010216 21702 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:21.008149 21687 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:21.031998 21680 raft_consensus.cc:695] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:21.008773 21558 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:21.012835 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 729) took 1353 ms. Trace:
I20250831 13:58:21.075706 21725 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.075604 21680 consensus_queue.cc:237] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.075368 20642 rpcz_store.cc:276] 0831 13:58:19.659064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.002981 (+1343917us) service_pool.cc:225] Handling call
0831 13:58:21.003322 (+ 341us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.005326 (+ 2004us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.007755 (+ 2429us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:21.009789 (+ 2034us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:21.012566 (+ 2777us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.012653 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.012780 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.077726 21701 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.062135 20650 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:21.026597 21584 consensus_meta.cc:287] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.813s user 0.002s sys 0.005s
I20250831 13:58:21.066346 20644 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:21.060056 21697 consensus_meta.cc:287] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 1.012s user 0.006s sys 0.013s
I20250831 13:58:20.967585 21501 consensus_queue.cc:237] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:21.064527 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.085760 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 734) took 1238 ms. Trace:
I20250831 13:58:21.023959 21630 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:21.101366 20650 rpcz_store.cc:276] 0831 13:58:19.847579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.061684 (+1214105us) service_pool.cc:225] Handling call
0831 13:58:21.061961 (+ 277us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.085429 (+ 23468us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.085684 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.087548 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 732) took 1424 ms. Trace:
I20250831 13:58:21.002987 21634 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:21.008936 21473 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:20.989737 21547 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.096848 20541 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.058372 21659 raft_consensus.cc:738] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:21.087174 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:21.062088 21585 consensus_queue.cc:260] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:21.102149 20647 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 1.
I20250831 13:58:21.107506 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 880) took 1426 ms. Trace:
I20250831 13:58:21.199288 20574 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:21.199499 20568 rpcz_store.cc:276] 0831 13:58:19.681170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.892834 (+211664us) service_pool.cc:225] Handling call
0831 13:58:19.918514 (+ 25680us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:19.936169 (+ 17655us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.107316 (+1171147us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b54001a68d0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:21.107450 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3240192}
W20250831 13:58:21.139808 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:21.104393 20644 rpcz_store.cc:276] 0831 13:58:19.663243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.016015 (+1352772us) service_pool.cc:225] Handling call
0831 13:58:21.066141 (+ 50126us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.087294 (+ 21153us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.087475 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:21.125613 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:21.201063 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:21.201093 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 104383us
W20250831 13:58:21.163892 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 99655.1us
I20250831 13:58:21.202152 20398 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54880 (request call id 33) took 5144 ms. Trace:
I20250831 13:58:21.106184 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 462) took 1464 ms. Trace:
I20250831 13:58:21.201900 20541 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.130825 20650 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:21.203596 20647 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:21.256814 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 54991.4us
I20250831 13:58:21.256973 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 475) took 1160 ms. Trace:
W20250831 13:58:21.178855 21643 consensus_meta.cc:287] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.056s user 0.012s sys 0.000s
I20250831 13:58:21.258466 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 744) took 1317 ms. Trace:
W20250831 13:58:21.258812 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 54306.3us
W20250831 13:58:21.135069 21733 log.cc:927] Time spent T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20250831 13:58:21.257591 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 684) took 2478 ms. Trace:
I20250831 13:58:21.267755 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 745) took 1281 ms. Trace:
I20250831 13:58:21.268221 20644 rpcz_store.cc:276] 0831 13:58:20.003508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.201083 (+1197575us) service_pool.cc:225] Handling call
0831 13:58:21.267132 (+ 66049us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.267350 (+ 218us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.267388 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:21.267470 (+ 82us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:21.267513 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.267581 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.267691 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.264024 20571 rpcz_store.cc:276] 0831 13:58:18.781884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.311047 (+529163us) service_pool.cc:225] Handling call
0831 13:58:19.311329 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:19.325603 (+ 14274us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:19.325642 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:19.325684 (+ 42us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:19.325715 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.257386 (+1931671us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b54001fb154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:21.257437 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.257543 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32311168}
I20250831 13:58:21.085222 21655 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.262020 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:21.202531 20398 rpcz_store.cc:276] 0831 13:58:16.051279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:16.051554 (+ 275us) service_pool.cc:225] Handling call
0831 13:58:21.195001 (+5143447us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2445668,"reported_tablets":1}
I20250831 13:58:21.269626 21676 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:21.273716 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:21.272485 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:21.277349 20617 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:21.261389 20650 rpcz_store.cc:276] 0831 13:58:19.940577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.130280 (+1189703us) service_pool.cc:225] Handling call
0831 13:58:21.130629 (+ 349us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.256291 (+125662us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.258413 (+ 2122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.159315 21706 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:21.133808 21618 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:21.192281 21614 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.254271 21681 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:21.163630 21687 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:21.279704 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:21.257380 20647 rpcz_store.cc:276] 0831 13:58:20.096184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.201859 (+1105675us) service_pool.cc:225] Handling call
0831 13:58:21.202174 (+ 315us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.256753 (+ 54579us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.256920 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.287643 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:21.203102 20651 rpcz_store.cc:276] 0831 13:58:19.641688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.653994 (+ 12306us) service_pool.cc:225] Handling call
0831 13:58:21.105338 (+1451344us) spinlock_profiling.cc:243] Waited 7.03 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:21.105606 (+ 268us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.105801 (+ 195us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.105839 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:21.105903 (+ 64us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:21.105941 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.106010 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.106127 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16158464}
I20250831 13:58:21.289136 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.289564 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:21.269661 21671 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:21.279992 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.086464 21702 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.282114 20574 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:21.259482 21615 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:21.088606 21584 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.293985 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 477) took 1140 ms. Trace:
I20250831 13:58:21.294903 21584 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.276970 21696 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:21.157620 21634 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:21.191972 21720 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.291560 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:21.292148 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:21.284252 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:21.302651 20576 raft_consensus.cc:2391] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:21.297818 20571 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.277841 21723 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:21.283877 21651 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:21.286859 21496 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:21.311242 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.102695 21697 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.305773 21722 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.309396 20576 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 13:58:21.316910 20576 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.318459 20574 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.309698 20571 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:21.272083 21643 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.106807 21672 consensus_queue.cc:237] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.290489 21585 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:21.279234 21680 consensus_queue.cc:1046] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20250831 13:58:21.107048 21562 consensus_queue.cc:237] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:21.312928 21491 consensus_meta.cc:287] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.762s user 0.002s sys 0.009s
I20250831 13:58:21.328529 20571 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.291214 21655 consensus_queue.cc:1046] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20250831 13:58:21.329965 20643 raft_consensus.cc:2391] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:21.327204 20576 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.296186 21687 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:21.332398 20497 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 13:58:21.333606 21722 raft_consensus.cc:3053] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:21.294356 20651 rpcz_store.cc:276] 0831 13:58:20.153556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.293067 (+1139511us) service_pool.cc:225] Handling call
0831 13:58:21.293409 (+ 342us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.293630 (+ 221us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.293667 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:21.293721 (+ 54us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:21.293752 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.293816 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.293935 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.294991 21671 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:21.341095 21740 log.cc:927] Time spent T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
I20250831 13:58:21.330037 21716 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:21.315357 21651 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.299379 21659 consensus_queue.cc:260] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.345968 20651 raft_consensus.cc:1273] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:21.326238 21741 consensus_queue.cc:1046] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20250831 13:58:21.337407 21722 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:21.343021 21473 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.330510 21491 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.359385 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 480) took 1034 ms. Trace:
I20250831 13:58:21.361564 21702 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.363263 20650 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:21.309486 21618 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.365295 21508 raft_consensus.cc:3053] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:21.369557 21508 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:21.362257 20651 rpcz_store.cc:276] 0831 13:58:20.324692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.341268 (+1016576us) service_pool.cc:225] Handling call
0831 13:58:21.341785 (+ 517us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.353720 (+ 11935us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.359323 (+ 5603us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:21.371908 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:21.372727 20648 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 728) took 1713 ms (client timeout 1642 ms). Trace:
W20250831 13:58:21.373060 20648 rpcz_store.cc:269] 0831 13:58:19.658833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.669896 (+ 11063us) service_pool.cc:225] Handling call
0831 13:58:21.371765 (+1701869us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:21.372660 (+ 895us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11016448}
I20250831 13:58:21.374191 20648 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:21.364447 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:21.376541 20493 raft_consensus.cc:2391] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:21.376714 21681 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:21.377791 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:21.366071 21697 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:21.316228 21696 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.371567 21710 raft_consensus.cc:3053] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:21.384207 20648 raft_consensus.cc:1238] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:21.324081 21723 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.384977 21508 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:21.390086 20648 raft_consensus.cc:3053] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:21.387131 20570 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:21.390412 20648 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:21.391366 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:21.396883 20646 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46464 (request call id 16) took 1621 ms (client timeout 1861 ms). Trace:
W20250831 13:58:21.397279 20646 rpcz_store.cc:269] 0831 13:58:19.775548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.547060 (+771512us) service_pool.cc:225] Handling call
0831 13:58:21.391193 (+844133us) spinlock_profiling.cc:243] Waited 6.82 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:21.396825 (+ 5632us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15693568}
I20250831 13:58:21.400089 21174 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.370285 21659 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:21.388749 20651 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:21.398638 20646 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:21.388177 21720 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:21.405107 21634 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:21.410512 21634 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:21.412989 21723 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.413376 20651 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.413903 21634 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:21.403306 21739 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:21.401173 21710 raft_consensus.cc:738] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:21.403403 20648 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.415143 21508 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:21.426292 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:58:21.423502 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 160654us
W20250831 13:58:21.423066 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 161807us
I20250831 13:58:21.427009 21210 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:21.431727 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 874) took 2098 ms. Trace:
I20250831 13:58:21.434717 20502 rpcz_store.cc:276] 0831 13:58:19.332923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.698052 (+365129us) service_pool.cc:225] Handling call
0831 13:58:21.431477 (+1733425us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b5400185bd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:21.431656 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1132099328}
I20250831 13:58:21.435251 21618 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:21.432559 20498 raft_consensus.cc:1238] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.451879 20466 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:21.456532 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:21.427976 21643 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.427258 21687 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:21.508013 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 56218.4us
I20250831 13:58:21.508097 21738 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:21.415303 21671 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.435933 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:21.507437 21618 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:21.508948 21582 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:21.466933 21634 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:21.416807 21722 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.519408 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:21.519112 21710 consensus_queue.cc:260] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.515488 21651 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:21.527866 21696 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:21.470798 21742 consensus_queue.cc:1046] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
W20250831 13:58:21.535760 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:21.524500 21738 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:21.536489 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 264680us
I20250831 13:58:21.521567 21491 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.515197 21642 consensus_queue.cc:1046] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20250831 13:58:21.519740 21565 consensus_queue.cc:1046] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20250831 13:58:21.520746 21582 consensus_queue.cc:1046] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20250831 13:58:21.524855 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.529628 21722 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:21.555253 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:21.520177 21584 consensus_queue.cc:1046] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20250831 13:58:21.551105 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:21.518561 21618 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.556280 20576 raft_consensus.cc:2391] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:21.521272 21634 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:21.543182 21561 consensus_meta.cc:287] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.995s user 0.007s sys 0.005s
I20250831 13:58:21.567114 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 736) took 1382 ms. Trace:
I20250831 13:58:21.561084 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:21.567497 20573 rpcz_store.cc:276] 0831 13:58:20.185080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.417184 (+232104us) service_pool.cc:225] Handling call
0831 13:58:20.807755 (+390571us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.807921 (+ 166us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:20.807950 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:20.807993 (+ 43us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:20.808021 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.808072 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.566961 (+758889us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54000e7250. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:21.567059 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":897539456}
I20250831 13:58:21.568609 20576 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:21.562409 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:21.565552 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:21.571892 21584 raft_consensus.cc:3053] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:21.572153 21738 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.573694 21656 raft_consensus.cc:3053] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:21.574103 21656 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:21.576009 20568 raft_consensus.cc:1238] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.577287 20499 raft_consensus.cc:1273] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:21.575208 21559 raft_consensus.cc:3053] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:21.574959 21656 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.572283 21584 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:21.598742 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.580224 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.600346 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:21.556267 21671 consensus_queue.cc:1046] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20250831 13:58:21.585655 21559 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:21.613504 21745 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:21.614243 21745 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.619961 21745 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.610363 21559 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.621789 21559 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:21.630607 21634 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.633271 21618 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:21.634853 21584 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.636596 21584 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:21.655750 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:21.667132 20643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 474) took 1643 ms (client timeout 2044 ms). Trace:
W20250831 13:58:21.667590 20643 rpcz_store.cc:269] 0831 13:58:20.023470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.833826 (+810356us) service_pool.cc:225] Handling call
0831 13:58:21.666911 (+833085us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b5400373a54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:21.667067 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1135941376}
W20250831 13:58:21.673745 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:21.613082 21656 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:21.676086 20642 raft_consensus.cc:1238] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.679790 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 855) took 2357 ms. Trace:
I20250831 13:58:21.680173 20501 rpcz_store.cc:276] 0831 13:58:19.322570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.813250 (+490680us) service_pool.cc:225] Handling call
0831 13:58:20.216913 (+403663us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.217082 (+ 169us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:20.217108 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:20.217147 (+ 39us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:20.217177 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.217233 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:21.679599 (+1462366us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b54001c2850. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:21.679723 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":927352064}
I20250831 13:58:21.681838 20501 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:21.672612 21634 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:21.698905 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:21.699647 20501 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:21.701411 20467 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:21.697014 21706 raft_consensus.cc:3053] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:21.715152 20467 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:21.719789 21565 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:21.723632 21565 raft_consensus.cc:695] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:21.694268 21634 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.735148 21565 consensus_queue.cc:237] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.736739 20501 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:21.714969 21706 raft_consensus.cc:738] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:21.739578 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 428432us
I20250831 13:58:21.738307 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:21.741750 20501 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:21.741017 20617 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.743820 20617 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.724673 21603 consensus_queue.cc:1046] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20250831 13:58:21.729668 21561 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
W20250831 13:58:21.751530 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 241672us
I20250831 13:58:21.748246 21648 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:21.755328 21702 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:21.745363 21651 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:21.743108 21696 consensus_queue.cc:1046] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
W20250831 13:58:21.757594 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:21.763340 20541 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.757797 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.764775 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:21.765249 20649 raft_consensus.cc:2366] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5.
W20250831 13:58:21.765791 20649 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 727) took 2107 ms (client timeout 1592 ms). Trace:
W20250831 13:58:21.766197 20649 rpcz_store.cc:269] 0831 13:58:19.658547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.668258 (+ 9711us) service_pool.cc:225] Handling call
0831 13:58:21.757627 (+2089369us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:21.765696 (+ 8069us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14691584}
I20250831 13:58:21.766942 21528 raft_consensus.cc:3053] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:21.767307 21528 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:21.768091 21528 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:21.756505 20617 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.771395 21555 consensus_queue.cc:1035] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20250831 13:58:21.798036 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:21.758823 21603 consensus_queue.cc:1230] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer 212b66976f874aa682cae0fb77928786 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 13:58:21.811094 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:21.813122 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:21.813153 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 56718.4us
I20250831 13:58:21.815209 20617 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.810226 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:21.816944 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:21.818133 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 730) took 1634 ms. Trace:
I20250831 13:58:21.818513 20569 rpcz_store.cc:276] 0831 13:58:20.183284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.184411 (+ 1127us) service_pool.cc:225] Handling call
0831 13:58:20.200010 (+ 15599us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.616980 (+416970us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.817904 (+1200924us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b54003b5454. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:21.818076 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":859271552}
I20250831 13:58:21.819586 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 436) took 2976 ms. Trace:
I20250831 13:58:21.820449 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:21.807214 21635 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:21.821615 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.819509 20645 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:21.816912 20643 raft_consensus.cc:2391] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:21.962517 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:21.963929 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:21.965270 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.966416 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:21.967782 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 873) took 2635 ms. Trace:
I20250831 13:58:22.048914 20495 rpcz_store.cc:276] 0831 13:58:19.332695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.051164 (+718469us) service_pool.cc:225] Handling call
0831 13:58:20.095509 (+ 44345us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.878286 (+782777us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.967520 (+1089234us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b540021d9d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:21.967715 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1798807936}
I20250831 13:58:21.980512 21644 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:21.980724 21645 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 13:58:21.974659 20645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 723) took 2326 ms (client timeout 1908 ms). Trace:
I20250831 13:58:21.962248 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:21.971909 20543 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:21.808009 21540 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:21.986961 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:21.808198 21709 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:21.812340 21542 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:21.916430 21508 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:22.000185 20541 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.063256 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:22.001889 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.003111 20568 raft_consensus.cc:1238] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:22.004365 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 474811us
I20250831 13:58:22.013546 21755 consensus_queue.cc:1046] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250831 13:58:22.017321 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:22.048228 20501 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:21.969477 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 154387us
I20250831 13:58:21.811409 21501 consensus_queue.cc:1046] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.208s
I20250831 13:58:21.984849 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:22.054867 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 462) took 1733 ms. Trace:
I20250831 13:58:22.095842 20498 rpcz_store.cc:276] 0831 13:58:20.321267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.321393 (+ 126us) service_pool.cc:225] Handling call
0831 13:58:21.432252 (+1110859us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.438719 (+ 6467us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.019903 (+581184us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400185bd0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:22.054817 (+ 34914us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2531483520}
I20250831 13:58:21.820129 20496 rpcz_store.cc:276] 0831 13:58:18.844468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.057619 (+213151us) service_pool.cc:225] Handling call
0831 13:58:19.084796 (+ 27177us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:19.790828 (+706032us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:21.819359 (+2028531us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54001856d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:21.819527 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1459822976}
I20250831 13:58:22.058866 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:21.991461 21723 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:22.061681 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 110373us
I20250831 13:58:21.987099 21175 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.068591 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.100945 20501 raft_consensus.cc:2391] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:22.018810 21479 consensus_queue.cc:1046] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20250831 13:58:22.078591 21651 consensus_queue.cc:1046] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:22.079553 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:22.102602 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:22.079883 21565 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:22.080159 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:22.080750 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:22.085991 21672 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:22.086540 21680 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.089943 21501 raft_consensus.cc:3053] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.060503 21540 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.121810 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.062155 21727 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:22.106287 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.107038 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.107318 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 862) took 3474 ms. Trace:
I20250831 13:58:22.109295 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.123955 20498 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:22.067842 21753 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:22.117923 20502 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.041s sys 0.134s
W20250831 13:58:22.120750 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.124007 20575 rpcz_store.cc:276] 0831 13:58:18.633390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:18.711782 (+ 78392us) service_pool.cc:225] Handling call
0831 13:58:19.902876 (+1191094us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:19.903198 (+ 322us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:19.903245 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:19.928365 (+ 25120us) log.cc:844] Serialized 26 byte log entry
0831 13:58:20.144739 (+216374us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:20.156159 (+ 11420us) log.cc:844] Serialized 14 byte log entry
0831 13:58:20.158447 (+ 2288us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:20.158545 (+ 98us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:20.574466 (+415921us) raft_consensus.cc:1683] finished
0831 13:58:20.574488 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:22.107167 (+1532679us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540031e2d0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:22.107268 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9144,"prepare.queue_time_us":10817,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":5916,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":10378,"spinlock_wait_cycles":2738564992,"thread_start_us":184453,"threads_started":1,"wal-append.queue_time_us":285397,"wal-append.run_cpu_time_us":3133,"wal-append.run_wall_time_us":1266223}
I20250831 13:58:22.105631 21634 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:22.058353 20645 rpcz_store.cc:269] 0831 13:58:19.648704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:19.652789 (+ 4085us) service_pool.cc:225] Handling call
0831 13:58:21.764624 (+2111835us) spinlock_profiling.cc:243] Waited 8.72 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:21.949992 (+185368us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20051840}
I20250831 13:58:22.066247 21709 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:21.776979 21706 consensus_queue.cc:260] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.151409 20575 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:22.128985 20616 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.126708 20573 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.152720 20574 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:22.153504 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:22.127138 20502 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.611s user 0.044s sys 0.134s
I20250831 13:58:22.120676 21501 raft_consensus.cc:738] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:22.125797 21672 raft_consensus.cc:695] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:22.148626 21753 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.103668 21723 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:22.149242 21559 consensus_meta.cc:287] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.527s user 0.008s sys 0.002s
W20250831 13:58:22.186704 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 57785.7us
I20250831 13:58:22.121272 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 468) took 1790 ms. Trace:
I20250831 13:58:22.119974 21562 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.197835 20501 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:22.118032 21565 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.195647 21501 consensus_queue.cc:260] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.197134 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:22.187718 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 187635us
I20250831 13:58:22.186554 21651 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:22.201215 21562 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.204387 21723 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.205062 21540 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.206302 20646 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:22.206926 21559 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:22.207562 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.225174 21648 raft_consensus.cc:3053] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.223605 21672 consensus_queue.cc:237] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.241020 21648 raft_consensus.cc:738] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:22.198421 20494 rpcz_store.cc:276] 0831 13:58:20.328870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.448746 (+119876us) service_pool.cc:225] Handling call
0831 13:58:20.449051 (+ 305us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.876902 (+427851us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.109062 (+1232160us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b540009a1d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:22.119404 (+ 10342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":790805248}
I20250831 13:58:22.231750 21708 raft_consensus.cc:3053] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.243767 21708 raft_consensus.cc:738] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:22.244100 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.244518 21708 consensus_queue.cc:260] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.242173 21648 consensus_queue.cc:260] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.245795 21708 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.245992 21648 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.219341 20576 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.230803 21673 raft_consensus.cc:3053] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.233553 21457 raft_consensus.cc:3053] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.251377 21457 raft_consensus.cc:738] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:22.252161 21457 consensus_queue.cc:260] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.244732 20494 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:22.253513 21457 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.254029 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.254128 20648 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.237455 21725 raft_consensus.cc:3053] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.255182 21725 raft_consensus.cc:738] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:22.256135 21725 consensus_queue.cc:260] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.257330 21725 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.259696 21588 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.250370 21673 raft_consensus.cc:738] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:58:22.226379 20572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 728) took 2044 ms (client timeout 1642 ms). Trace:
W20250831 13:58:22.261015 20572 rpcz_store.cc:269] 0831 13:58:20.182501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.182672 (+ 171us) service_pool.cc:225] Handling call
0831 13:58:22.226111 (+2043439us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b54000e7754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:22.226284 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":638772992}
I20250831 13:58:22.236745 20650 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20250831 13:58:22.209522 21656 consensus_meta.cc:287] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.525s user 0.007s sys 0.002s
I20250831 13:58:22.239120 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 478) took 2029 ms. Trace:
I20250831 13:58:22.189785 21603 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250831 13:58:22.273870 20644 rpcz_store.cc:276] 0831 13:58:20.209815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.306912 (+1097097us) service_pool.cc:225] Handling call
0831 13:58:22.237599 (+930687us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:22.237796 (+ 197us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:22.237841 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:22.237895 (+ 54us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:22.237942 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.237995 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:22.238934 (+ 939us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54001ee950. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:22.239050 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2109791488}
I20250831 13:58:22.218469 21544 consensus_queue.cc:1046] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
W20250831 13:58:22.203505 21722 consensus_meta.cc:287] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.642s user 0.011s sys 0.000s
I20250831 13:58:22.281535 20648 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250831 13:58:22.227849 21584 consensus_meta.cc:287] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.591s user 0.007s sys 0.003s
I20250831 13:58:22.261822 20649 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.232069 21562 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.296864 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 481) took 1801 ms. Trace:
I20250831 13:58:22.296989 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 476) took 2170 ms. Trace:
I20250831 13:58:22.285439 20495 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:22.331125 20648 rpcz_store.cc:276] 0831 13:58:20.495498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.389745 (+894247us) service_pool.cc:225] Handling call
0831 13:58:21.389975 (+ 230us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:22.253939 (+863964us) spinlock_profiling.cc:243] Waited 828 ms on lock 0x7b58001c6ab0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc8f1003 00007fb9bc954577 00007fb9bc96cfa7 00007fb9bc96e79e 00007fb9bc978af9 00007fb9bc9793c6 00007fb9bc975d4f 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44
0831 13:58:22.296603 (+ 42664us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.296811 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1904425728}
I20250831 13:58:22.233595 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.332188 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.332458 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.333316 20648 raft_consensus.cc:2366] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5.
I20250831 13:58:22.333061 20644 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:22.240010 21753 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:22.297981 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 326148us
W20250831 13:58:22.329463 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 137252us
I20250831 13:58:22.331192 20647 rpcz_store.cc:276] 0831 13:58:20.127056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.291159 (+1164103us) service_pool.cc:225] Handling call
0831 13:58:21.291387 (+ 228us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.391936 (+100549us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.391979 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:21.392034 (+ 55us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:21.392078 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.296804 (+904726us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b540029fc54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:22.296842 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:22.296943 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230771968}
I20250831 13:58:22.255193 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.342713 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.343681 20495 raft_consensus.cc:2391] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:22.345675 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.346974 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.348579 20647 raft_consensus.cc:1273] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:22.348620 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.347920 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.349794 20647 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:22.352675 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:22.352840 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:22.353370 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:22.353615 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.297646 21501 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.354646 20617 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:22.357436 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:22.217245 21651 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:22.330482 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 50777.6us
I20250831 13:58:22.325204 21656 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:22.328251 21724 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.363970 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.335922 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:22.362928 20541 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.364681 21544 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:22.365655 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:22.369647 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:22.375044 21730 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20250831 13:58:22.377629 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 880) took 1376 ms. Trace:
I20250831 13:58:22.494330 20500 rpcz_store.cc:276] 0831 13:58:21.001169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.059997 (+ 58828us) service_pool.cc:225] Handling call
0831 13:58:21.060254 (+ 257us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.078516 (+ 18262us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.078544 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:21.078583 (+ 39us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:21.078624 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.377380 (+1298756us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b540021c5d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:22.377436 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:22.377559 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41600768}
I20250831 13:58:22.496315 20500 raft_consensus.cc:1238] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.497722 20500 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:22.381635 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 471) took 1954 ms. Trace:
I20250831 13:58:22.498250 20499 rpcz_store.cc:276] 0831 13:58:20.427127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.467842 (+ 40715us) service_pool.cc:225] Handling call
0831 13:58:20.472068 (+ 4226us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.577743 (+1105675us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.381348 (+803605us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002201d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:22.381552 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2540869888}
I20250831 13:58:22.383423 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 735) took 2533 ms. Trace:
I20250831 13:58:22.387264 20645 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.391016 21643 raft_consensus.cc:3053] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:22.394359 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 899) took 1581 ms. Trace:
I20250831 13:58:22.395463 21549 raft_consensus.cc:3053] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.510159 20570 rpcz_store.cc:276] 0831 13:58:20.813437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.848122 (+ 34685us) service_pool.cc:225] Handling call
0831 13:58:20.855176 (+ 7054us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.393511 (+538335us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.394078 (+1000567us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b54001a63d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:22.394290 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1222993152}
I20250831 13:58:22.405462 20501 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.406700 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 877) took 1406 ms. Trace:
I20250831 13:58:22.408946 20574 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.518402 20497 rpcz_store.cc:276] 0831 13:58:21.000283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.058995 (+ 58712us) service_pool.cc:225] Handling call
0831 13:58:21.059318 (+ 323us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:21.334828 (+275510us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.406443 (+1071615us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400186fd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:22.406627 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":627545344}
I20250831 13:58:22.520584 20501 raft_consensus.cc:1238] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.411407 21528 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:22.527444 20501 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:22.464203 21565 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.417786 21547 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:22.419639 21559 raft_consensus.cc:3053] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.425046 21738 raft_consensus.cc:3053] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.361804 20494 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:22.454002 21473 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.458586 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:22.460897 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 244005us
I20250831 13:58:22.463081 20576 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:22.464905 20573 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.356750 21580 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:22.363555 21753 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.288858 21673 consensus_queue.cc:260] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.462913 21544 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:22.471629 21737 raft_consensus.cc:3053] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:22.472371 20495 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.516484 21549 raft_consensus.cc:738] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:22.503245 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 167406us
W20250831 13:58:22.556903 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:22.504252 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 141428us
W20250831 13:58:22.501410 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 147550us
I20250831 13:58:22.472909 20496 raft_consensus.cc:2391] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:22.529570 20647 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.364395 21762 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:22.416373 20646 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:22.538745 21559 raft_consensus.cc:738] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:22.515286 21546 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:22.547196 20500 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:22.509932 21643 raft_consensus.cc:738] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:22.550637 20499 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.573058 20466 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.527782 21748 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:22.462656 21656 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.367647 20617 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:22.488690 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 569157us
I20250831 13:58:22.552800 21737 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:22.559432 20541 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.472322 21724 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.533928 20501 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.543885 21738 raft_consensus.cc:738] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:22.552815 21673 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.554190 21544 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:22.569296 21503 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:22.581624 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 215496us
I20250831 13:58:22.500941 20642 rpcz_store.cc:276] 0831 13:58:19.850214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.076979 (+1226765us) service_pool.cc:225] Handling call
0831 13:58:21.673112 (+596133us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.683964 (+ 10852us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.383143 (+699179us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b5400373a50. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:22.383346 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1367753088}
I20250831 13:58:22.581279 20616 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.590858 20617 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:22.551237 21473 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.591953 20617 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:22.572577 21651 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:22.574123 21546 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:22.551848 21766 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.593129 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:22.597769 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.598173 20466 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.593986 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 896) took 2400 ms. Trace:
I20250831 13:58:22.559523 21753 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.551023 21767 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:22.559682 21580 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:22.579308 21561 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:22.597103 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 881) took 1595 ms. Trace:
I20250831 13:58:22.604350 20569 raft_consensus.cc:1273] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:22.578883 21709 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.594354 21349 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.605835 21562 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.592732 21599 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:22.598927 20577 rpcz_store.cc:276] 0831 13:58:20.193262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:20.663791 (+470529us) service_pool.cc:225] Handling call
0831 13:58:20.744398 (+ 80607us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:20.744668 (+ 270us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:20.770026 (+ 25358us) log.cc:844] Serialized 14 byte log entry
0831 13:58:22.426715 (+1656689us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b18002a9460. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba2da 00007fb9bc9809da
0831 13:58:22.593666 (+166951us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:22.593718 (+ 52us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:22.593754 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.593827 (+ 73us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:22.593932 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19157,"prepare.queue_time_us":24780,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":22411,"spinlock_wait_cycles":11539328,"thread_start_us":147518,"threads_started":1}
I20250831 13:58:22.602963 20466 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.589675 21645 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 13:58:22.603581 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 44254.9us
I20250831 13:58:22.570366 21493 consensus_queue.cc:1046] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250831 13:58:22.607436 21646 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:22.607718 21754 consensus_queue.cc:1046] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:22.596674 21555 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:22.596020 21588 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.596675 21546 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:22.611320 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.595311 21503 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.612200 20541 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.592504 21751 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.594900 21708 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:22.596956 21720 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:22.594900 21540 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:22.601366 21565 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.616528 20543 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.618156 20466 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.589828 21705 consensus_queue.cc:1046] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250831 13:58:22.603000 21676 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:22.605520 21618 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:22.614750 21735 consensus_queue.cc:1046] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20250831 13:58:22.587100 21549 consensus_queue.cc:260] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.603708 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 885) took 1601 ms. Trace:
I20250831 13:58:22.627645 21708 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.630504 21680 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:22.631006 21540 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.627467 21349 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:22.661378 21349 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:22.650935 21615 consensus_queue.cc:1035] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250831 13:58:22.662947 21583 raft_consensus.cc:3053] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:22.663420 21583 raft_consensus.cc:738] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:22.664336 21583 consensus_queue.cc:260] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.664952 21765 raft_consensus.cc:3053] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.665362 21765 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:22.665513 21583 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:22.620566 21774 consensus_queue.cc:1046] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20250831 13:58:22.627223 21751 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.605572 21767 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:22.571811 21762 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.611882 21645 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:22.633710 20466 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.610069 21761 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.635628 20573 raft_consensus.cc:1273] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:22.619027 21642 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:22.679199 21541 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.641885 21618 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:22.629711 21566 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.642951 21730 consensus_queue.cc:1046] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250831 13:58:22.688011 21566 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.610749 21561 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.692271 21561 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.643056 20493 rpcz_store.cc:276] 0831 13:58:21.005623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.538159 (+532536us) service_pool.cc:225] Handling call
0831 13:58:21.582659 (+ 44500us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:21.607897 (+ 25238us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:21.612639 (+ 4742us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:21.640226 (+ 27587us) log.cc:844] Serialized 26 byte log entry
0831 13:58:22.018602 (+378376us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:22.512617 (+494015us) log.cc:844] Serialized 14 byte log entry
0831 13:58:22.584160 (+ 71543us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.590221 (+ 6061us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:22.593163 (+ 2942us) raft_consensus.cc:1683] finished
0831 13:58:22.595850 (+ 2687us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:22.602013 (+ 6163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":161332,"prepare.queue_time_us":400849,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":69065,"raft.run_cpu_time_us":409,"raft.run_wall_time_us":6029,"thread_start_us":769285,"threads_started":2,"wal-append.queue_time_us":442219}
I20250831 13:58:22.680565 21582 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:22.610069 21737 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.699766 21706 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:22.700011 21737 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:22.653877 21709 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.643476 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.652889 21205 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 3 LEARNER]: Advancing to term 5
I20250831 13:58:22.663414 21349 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:22.624409 21588 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:22.667508 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.604390 21559 consensus_queue.cc:260] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.625995 21672 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.635118 21676 raft_consensus.cc:695] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:22.639642 21646 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.623018 21555 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.627980 20541 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:22.722319 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 94416.1us
I20250831 13:58:22.639225 21732 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:22.616407 21643 consensus_queue.cc:260] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.651243 21562 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.686400 20466 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:22.658835 21503 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.690435 21566 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:22.651561 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.705483 20498 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:22.642918 21720 raft_consensus.cc:695] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:22.653967 21626 consensus_queue.cc:1046] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20250831 13:58:22.680693 21645 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.694732 20398 catalog_manager.cc:5582] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: config changed from index 3 to 4, VOTER 44d163094add49049cc749547d891602 (127.18.233.131) evicted. New cstate: current_term: 1 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:22.702255 21618 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:22.774672 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 157551us
W20250831 13:58:22.775285 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 89013.6us
I20250831 13:58:22.635545 21738 consensus_queue.cc:260] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:22.754563 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.679705 21541 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:22.723636 21635 consensus_meta.cc:287] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.902s user 0.006s sys 0.004s
W20250831 13:58:22.762622 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.747892 21656 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.666182 21765 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:22.749881 21644 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:22.712921 21709 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.762953 21643 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:22.777231 20466 leader_election.cc:400] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5.
I20250831 13:58:22.788193 20466 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.789229 20466 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.780735 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.781952 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:22.790696 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.768297 21753 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.784854 20398 catalog_manager.cc:5582] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:22.776321 21724 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.794127 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.777521 21626 consensus_queue.cc:1230] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer 212b66976f874aa682cae0fb77928786 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 13:58:22.779021 21582 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:22.765913 21412 consensus_queue.cc:1035] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20250831 13:58:22.798763 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.785893 21644 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:22.805632 20398 catalog_manager.cc:5582] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:22.800271 20543 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.802377 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.803056 20647 raft_consensus.cc:1273] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:22.795986 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.810401 20500 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:22.783613 21635 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.757871 21562 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.807896 20576 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:22.805658 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.799616 20501 raft_consensus.cc:1273] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:22.791970 21748 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:22.795075 21563 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.815796 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:22.816030 20541 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.814507 20398 catalog_manager.cc:5582] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: config changed from index 2 to 3, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:22.819965 21349 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:22.820331 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.820488 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.820916 20574 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:22.821820 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.822456 20574 raft_consensus.cc:2391] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:22.816715 21681 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.006s
I20250831 13:58:22.824748 21649 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.803910 21765 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.830252 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.832894 20569 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:22.834263 20575 raft_consensus.cc:1238] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:22.816329 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.787431 21459 consensus_queue.cc:1046] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
W20250831 13:58:22.797049 21731 consensus_queue.cc:1162] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab7ade51fa134a1b814fab748f01d43d" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20250831 13:58:22.803961 21582 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:22.804394 21580 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:22.814392 21565 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:22.799345 21774 consensus_queue.cc:1046] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20250831 13:58:22.813558 21615 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.811630 21375 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20250831 13:58:22.701586 21585 consensus_queue.cc:1162] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "58aded50ab0d4609a053647eb11b1f48" responder_term: 4 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 13:58:22.812698 21644 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:22.837867 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.830862 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.854375 20500 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:22.845228 21582 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:22.820348 21349 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:22.786103 21761 consensus_queue.cc:1046] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
W20250831 13:58:22.850721 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:22.851848 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 47578.7us
I20250831 13:58:22.826399 21767 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:22.830715 21766 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.837409 21672 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:22.856966 20466 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:22.847558 21540 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.893160 20467 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:22.854666 20646 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
W20250831 13:58:22.897989 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.818141 21676 consensus_queue.cc:237] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.604666 20502 rpcz_store.cc:276] 0831 13:58:21.001556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.509395 (+507839us) service_pool.cc:225] Handling call
0831 13:58:21.510919 (+ 1524us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:21.511106 (+ 187us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:22.155990 (+644884us) log.cc:844] Serialized 14 byte log entry
0831 13:58:22.596793 (+440803us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:22.596848 (+ 55us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:22.596888 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:22.596950 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:22.597057 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41673,"prepare.queue_time_us":820084,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":3541,"raft.queue_time_us":42321,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":4084,"spinlock_wait_cycles":9295232,"thread_start_us":1038123,"threads_started":2,"wal-append.queue_time_us":439357}
I20250831 13:58:22.928505 21705 raft_consensus.cc:3053] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:22.896520 21617 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:22.862706 21349 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:22.872669 21588 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.944590 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.945288 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.863263 21751 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:22.903115 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.898571 21646 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:22.877132 21642 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20250831 13:58:22.957978 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.959146 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.959398 20493 raft_consensus.cc:1273] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:22.959897 20646 raft_consensus.cc:2391] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:22.960427 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:22.960775 21457 consensus_meta.cc:287] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.706s user 0.008s sys 0.003s
I20250831 13:58:22.961239 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:22.917078 21555 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.961853 20502 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:22.949493 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:22.961709 21544 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.963052 21555 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:22.963557 21555 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:22.963696 21544 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:22.964020 21555 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:22.876752 21672 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:22.965380 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 44324.3us
I20250831 13:58:22.944183 21617 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.915469 21767 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:22.956142 21725 consensus_meta.cc:287] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.698s user 0.007s sys 0.005s
I20250831 13:58:22.928912 21609 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:22.956544 21541 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.957365 21562 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:22.960886 21785 log.cc:927] Time spent T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20250831 13:58:22.817862 21720 consensus_queue.cc:237] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.948354 20647 raft_consensus.cc:2391] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:22.962103 20643 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:22.996649 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:22.996722 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:22.945211 20644 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:22.896741 21762 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.998037 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.998770 20647 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:22.999768 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.000948 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.001571 20647 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
W20250831 13:58:22.973237 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 157489us
I20250831 13:58:22.851891 21722 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20250831 13:58:22.877008 21463 consensus_queue.cc:1046] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20250831 13:58:22.932219 21745 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:22.981036 21656 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:22.984906 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.023737 20502 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:22.985273 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 167356us
I20250831 13:58:22.915894 21742 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20250831 13:58:22.992866 21651 raft_consensus.cc:3053] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:22.976050 20496 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:22.955616 21782 log.cc:927] Time spent T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20250831 13:58:23.005759 20541 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:22.928956 21705 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.008626 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:22.998667 21754 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20250831 13:58:22.996547 21176 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.002880 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.022312 21770 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.068476 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 776) took 1002 ms. Trace:
W20250831 13:58:23.027671 21501 consensus_meta.cc:287] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.666s user 0.007s sys 0.002s
I20250831 13:58:23.019310 21672 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:22.982014 21767 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:22.996274 21706 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.025103 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:22.983323 21709 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.081815 21767 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:23.028723 20616 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:23.082594 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 53955us
I20250831 13:58:23.082819 21707 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:23.083312 20493 raft_consensus.cc:2408] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:23.083280 21707 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:23.083814 21707 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.032338 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:22.979063 21724 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20250831 13:58:23.003736 21566 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.081606 21580 raft_consensus.cc:3053] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:23.049099 20500 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:23.022728 21745 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.054256 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.054701 21617 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.042249 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.093263 20573 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:22.980085 21503 consensus_queue.cc:1046] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20250831 13:58:23.030037 21651 raft_consensus.cc:738] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.059517 21644 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.060600 21705 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.061400 21737 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:23.061342 21473 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.040014 21787 consensus_queue.cc:1046] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
W20250831 13:58:23.062449 21528 consensus_meta.cc:287] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.524s user 0.008s sys 0.000s
I20250831 13:58:23.063977 21549 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.068864 20575 rpcz_store.cc:276] 0831 13:58:22.066451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:22.154451 (+ 88000us) service_pool.cc:225] Handling call
0831 13:58:22.185163 (+ 30712us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:22.834608 (+649445us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:23.068265 (+233657us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b54002ff654. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:23.068426 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1492390272}
I20250831 13:58:22.997010 21563 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:23.070181 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:23.072597 21349 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.984946 21609 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.075927 20466 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:23.127710 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 51868.9us
I20250831 13:58:23.128538 20466 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:23.079916 21582 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.076968 20576 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:23.084970 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 498) took 1158 ms. Trace:
I20250831 13:58:23.132712 20643 rpcz_store.cc:276] 0831 13:58:21.926241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:21.966158 (+ 39917us) service_pool.cc:225] Handling call
0831 13:58:23.084812 (+1118654us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b540029f754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:23.084923 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2287542144}
I20250831 13:58:23.129897 20466 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:23.133879 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:23.026985 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:23.026952 20543 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.090173 20502 raft_consensus.cc:2391] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:23.152278 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 125394us
I20250831 13:58:23.090971 21745 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.085034 20644 raft_consensus.cc:2374] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2. Re-sending same reply.
I20250831 13:58:23.104326 21555 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.104810 21205 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 LEARNER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:23.104862 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.090934 21566 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.039443 21674 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20250831 13:58:23.105609 21503 raft_consensus.cc:3053] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:23.105679 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:23.089108 21770 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:23.106515 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.106786 21651 consensus_queue.cc:260] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.132422 21709 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.200035 21646 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.088935 21580 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.091044 21706 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.111245 21705 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:23.116637 20568 raft_consensus.cc:2408] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250831 13:58:23.030455 21656 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:22.877224 21766 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:23.126252 21715 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.145391 21642 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:23.148813 20495 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:23.131995 21546 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:23.149852 21582 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.084369 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.175004 20500 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.199796 20466 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:23.202593 20543 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.148214 21645 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:23.150877 21673 consensus_meta.cc:287] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.562s user 0.007s sys 0.003s
W20250831 13:58:23.199012 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:23.208943 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.273557 20644 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
W20250831 13:58:23.212458 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 231464us
I20250831 13:58:23.128037 21643 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:23.129235 21544 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.133109 21767 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.109279 21644 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.211269 20642 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 5. Ops: []
I20250831 13:58:23.216723 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:23.284833 21767 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.285578 21207 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:23.217352 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.286813 21767 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.286996 20494 raft_consensus.cc:2391] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:23.288154 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.288915 20494 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:23.289687 21766 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.220078 21555 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.290197 21766 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.125806 21547 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
I20250831 13:58:23.290724 21770 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.213315 21709 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:23.221940 20617 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.222095 21565 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:23.221940 21501 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:23.223276 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:23.297827 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 74649.3us
I20250831 13:58:23.201202 21528 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.213497 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.252355 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.204631 21788 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.205492 21491 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.203233 21708 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.203502 21503 raft_consensus.cc:738] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:23.266439 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:23.128772 21563 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
W20250831 13:58:23.267093 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 67418.2us
W20250831 13:58:23.267176 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 64700.8us
I20250831 13:58:23.109175 21724 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.257498 21651 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.288175 20646 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:23.269443 21556 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:23.269619 21715 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.291957 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.272070 21674 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:23.310859 21732 consensus_meta.cc:287] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.556s user 0.009s sys 0.000s
W20250831 13:58:23.295472 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 73605.6us
I20250831 13:58:23.295670 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:23.298802 20467 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.297116 21709 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.302500 21596 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.300930 21528 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.290548 21555 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.282894 21544 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.307094 21491 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.309728 20466 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:23.412007 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 102387us
I20250831 13:58:23.311857 21724 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.312742 21582 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.313187 20497 raft_consensus.cc:1273] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:23.313439 21501 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.280407 20570 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:23.293262 21774 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.317363 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.334502 20500 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:23.294555 21644 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.297662 21547 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:23.270049 21787 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.401775 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:23.405079 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 106350us
I20250831 13:58:23.404712 20617 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:23.406793 20502 raft_consensus.cc:2391] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:23.302800 21788 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:23.398236 21618 consensus_meta.cc:287] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.612s user 0.003s sys 0.010s
I20250831 13:58:23.431018 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.255740 21646 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:23.408488 21596 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.404901 21556 raft_consensus.cc:695] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:23.419025 21562 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.399343 21473 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.309805 21643 consensus_queue.cc:1046] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.106s
I20250831 13:58:23.428218 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:23.428998 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.411265 21732 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:23.429772 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.430109 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.403146 21349 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.431497 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.432523 21609 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.433225 21720 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:23.437984 21596 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.414907 21747 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.411821 21528 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:23.424247 21798 log.cc:927] Time spent T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.098s user 0.000s sys 0.001s
I20250831 13:58:23.420985 20643 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:23.422250 21656 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.444536 20650 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:23.448053 21473 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.445371 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.457903 20495 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:23.430874 21774 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:23.418972 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:23.416172 21555 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.433699 21787 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.450935 20617 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.426883 21582 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.439771 21562 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.453910 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.446161 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.466893 20500 raft_consensus.cc:2391] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:23.443049 21648 raft_consensus.cc:3053] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:23.468817 21648 raft_consensus.cc:738] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.469055 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.469599 21648 consensus_queue.cc:260] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:23.470199 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 426847us
I20250831 13:58:23.450172 21720 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.470885 21720 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:23.472296 21582 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.456658 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:23.433706 21644 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:23.476193 21491 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.453045 21747 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.476792 21747 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.477267 21644 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:23.478307 20501 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:23.478983 21646 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:23.470526 21648 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.481609 21738 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.483489 20500 raft_consensus.cc:1238] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:23.467042 21562 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.487255 21563 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.488230 21715 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.489099 21544 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.489343 21501 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.452224 21803 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:23.490644 21788 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.465889 21737 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:23.492852 21787 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.492941 21707 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:23.459424 21546 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:23.495612 21724 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.435902 21618 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.468043 21556 consensus_queue.cc:237] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:23.497032 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 46203.9us
I20250831 13:58:23.468242 21473 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.502267 20495 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:23.463114 21656 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.455425 21706 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.515765 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.514413 20572 raft_consensus.cc:1238] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:23.507469 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:23.518014 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:23.513916 20576 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:23.520179 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.510520 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.522531 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.523797 20495 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:23.525074 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.525807 20495 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:23.533862 21707 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:23.534411 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.465018 21645 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.536654 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.536907 20501 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:23.537650 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.538617 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.539481 20575 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:23.509320 21787 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:23.540983 21787 raft_consensus.cc:695] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.541275 20572 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:23.541894 21787 consensus_queue.cc:237] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.542919 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:23.544567 21463 consensus_queue.cc:1046] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:23.545269 21463 consensus_queue.cc:1230] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [NON_LEADER]: Peer 58aded50ab0d4609a053647eb11b1f48 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 13:58:23.546188 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.547060 21554 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:23.502447 21546 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:23.547323 20617 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.547504 21554 raft_consensus.cc:695] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.547837 21546 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:23.548159 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.548907 20569 raft_consensus.cc:2391] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:23.548805 20576 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:23.548475 21554 consensus_queue.cc:237] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.489192 21745 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.549762 21546 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:23.551043 21745 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.550966 20494 raft_consensus.cc:2366] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:23.505507 20645 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:23.548424 21770 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:23.552057 21770 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.552410 21770 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.552448 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.550956 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:23.553675 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.509848 21730 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.547516 21596 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.548161 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.558879 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.559264 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.562667 20573 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:23.562942 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 132650us
I20250831 13:58:23.555023 21730 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.580708 20500 raft_consensus.cc:2391] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:23.582559 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.583375 21774 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.583602 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.584244 20569 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:23.585059 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:23.585958 21210 raft_consensus.cc:2391] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
I20250831 13:58:23.587087 21555 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.588119 21674 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:23.589085 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.560755 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.592200 20645 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:23.593310 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.537827 21656 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.578179 21645 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.596735 21645 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.594667 21528 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.597095 21645 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.598448 21528 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:23.595374 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:23.585146 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.583371 20573 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:23.596377 21656 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.603370 21730 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.603380 20577 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:23.604629 20466 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.583606 21596 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.534705 21783 raft_consensus.cc:3053] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:23.607139 21783 raft_consensus.cc:738] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.608814 21618 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.613385 20541 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.615916 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.624231 20575 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:23.618554 20541 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:23.625959 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:23.636935 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.637370 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.489125 21580 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.640661 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.645351 20650 raft_consensus.cc:2391] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:23.641028 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.625250 21738 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:23.612490 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.625504 21737 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:23.617985 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.655545 20616 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.655714 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.656306 20650 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:23.656934 20616 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.657974 20616 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.581238 21747 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.640223 21588 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.659478 20543 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.661607 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.666945 20543 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:23.667265 20572 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:23.662086 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.638115 21656 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:23.664227 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.668054 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.664669 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.583707 21802 consensus_queue.cc:1046] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.066s
I20250831 13:58:23.639007 21724 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:23.684049 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.684612 20575 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:23.684624 20569 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:23.686658 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.686789 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.687387 20574 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:23.687474 20575 raft_consensus.cc:2391] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:23.635991 21709 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.689764 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.690336 20573 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:23.690480 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.685700 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:23.688539 20543 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.692512 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.693531 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.695605 21722 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.696084 21722 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.697073 21503 consensus_queue.cc:260] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.697980 21722 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.699910 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.700404 21585 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:23.667901 21618 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:23.701723 20617 leader_election.cc:400] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:23.683759 21546 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:23.702054 20617 leader_election.cc:403] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:23.702320 21546 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:23.703016 20494 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:23.665557 21751 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 13:58:23.700500 21797 log.cc:927] Time spent T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
I20250831 13:58:23.665316 21787 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20250831 13:58:23.701983 21720 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:23.704874 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:23.705217 20494 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:23.702816 21546 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:23.714560 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.715346 20569 raft_consensus.cc:2408] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250831 13:58:23.649363 21738 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:23.707224 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.669756 21656 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.666217 21588 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.681690 21737 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.712555 21618 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.731338 21747 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.732627 21709 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.733376 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.734123 20496 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:23.734591 21720 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:23.734956 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.735071 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.735685 20496 raft_consensus.cc:2391] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:23.735847 20502 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:23.736936 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.737600 20502 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:23.698007 21503 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.712193 21491 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.741420 21491 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.704005 21473 consensus_queue.cc:1046] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20250831 13:58:23.743038 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:23.743382 21491 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.743400 21473 raft_consensus.cc:3053] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:23.744297 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.744685 21473 raft_consensus.cc:738] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:23.745456 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.746054 20642 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:23.745709 21473 consensus_queue.cc:260] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.687255 21705 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:23.716864 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.721385 20568 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:23.721684 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.720561 21375 raft_consensus.cc:3053] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:23.722445 21656 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:23.724901 20644 raft_consensus.cc:2391] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:23.724833 20616 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:23.725966 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:23.728186 20498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 895) took 1654 ms (client timeout 1706 ms). Trace:
I20250831 13:58:23.727658 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.724714 21588 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.762017 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.763890 20643 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:23.764752 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:23.765250 20643 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.714267 21710 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.751417 20466 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.762992 20541 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:23.771528 20467 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:23.720890 21738 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:23.753135 21375 raft_consensus.cc:738] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.758306 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.760102 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 495) took 1798 ms. Trace:
W20250831 13:58:23.782793 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 58040.2us
I20250831 13:58:23.782708 20571 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:23.782655 20497 rpcz_store.cc:276] 0831 13:58:21.961625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:22.519361 (+557736us) service_pool.cc:225] Handling call
0831 13:58:22.519572 (+ 211us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:23.415943 (+896371us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:23.759798 (+343855us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54001621d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:23.760030 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1824383232}
I20250831 13:58:23.783636 20616 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:23.767769 20498 rpcz_store.cc:269] 0831 13:58:22.073952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:22.124866 (+ 50914us) service_pool.cc:225] Handling call
0831 13:58:23.728043 (+1603177us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b5400186ad4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:23.728136 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":390764288}
I20250831 13:58:23.783841 20541 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:23.785362 20541 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:23.769558 20499 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:23.730589 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.730057 21766 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:23.752287 20493 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:23.791220 20570 raft_consensus.cc:1238] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:23.779433 20467 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.786227 20616 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.788982 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.790019 21705 consensus_queue.cc:1046] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:23.788726 21674 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:23.791595 21735 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 13:58:23.795836 21799 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.487s user 0.000s sys 0.003s
I20250831 13:58:23.797363 21645 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.798962 20645 raft_consensus.cc:2366] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3.
I20250831 13:58:23.799346 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:23.799875 21566 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.800412 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:23.801426 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.801952 20644 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:23.802662 21580 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:23.802804 21673 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.803407 21803 raft_consensus.cc:3053] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:23.803365 21673 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.803880 21803 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:23.804697 21803 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:23.805295 21673 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.805975 21803 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:23.807436 21645 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:23.791069 21644 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:23.811292 21559 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.811463 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.812222 20501 raft_consensus.cc:2391] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:23.814139 21599 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:23.814507 21599 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.814881 21599 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:23.815488 21546 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:23.817747 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44784.4us
I20250831 13:58:23.818466 20467 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.819106 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.820863 20466 leader_election.cc:400] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3.
I20250831 13:58:23.821267 20466 leader_election.cc:403] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:23.823805 20577 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:23.746999 21562 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.826185 20616 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.830214 20541 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.833221 20467 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:23.787179 21565 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:23.787930 21747 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.824990 21582 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.791198 21659 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.849803 20543 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:23.850960 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.851908 20577 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:23.852164 20543 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:23.853998 20543 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:23.855334 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:23.856581 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.861208 20543 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.861721 20643 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:23.860420 21732 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:23.839025 21459 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:23.838713 21802 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250831 13:58:23.879294 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 84905.5us
I20250831 13:58:23.879889 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:23.879958 20617 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.880544 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.838171 21473 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:23.849542 21582 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.837199 21709 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.842067 21731 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:23.837849 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:23.860822 21174 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:23.865907 20543 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.866237 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.897531 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.867138 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.843683 21674 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:23.842504 21565 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.875466 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:23.825501 20646 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:23.881165 20497 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:23.846333 21561 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:23.847616 21747 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.885603 20571 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:23.887534 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:23.853338 21659 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.860575 21585 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:23.897701 20573 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.867818 20616 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:23.898638 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:23.899477 20541 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.864161 21528 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:23.875957 21673 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:23.863984 21732 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.876190 21463 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:23.904423 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:23.882376 21546 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.902976 20467 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:23.908486 20617 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:23.910926 21596 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.875444 21730 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20250831 13:58:23.901338 21580 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.918187 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:23.983836 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 116103us
I20250831 13:58:23.906765 21766 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:23.905539 21565 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:23.879120 21508 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.887228 21582 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.891238 21731 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:23.971490 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 68667.7us
I20250831 13:58:23.979848 21724 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:23.966650 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.001029 20643 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:23.910050 21561 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.980134 21751 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.884327 21556 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.922183 21747 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.918066 21770 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.993477 21349 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:23.929880 21651 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:23.929297 21585 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.930433 21528 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:23.965919 21503 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.965930 21732 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:23.924062 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 82308.9us
I20250831 13:58:23.930130 21727 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:23.901587 21656 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.970782 21609 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
W20250831 13:58:23.995728 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 87330.6us
W20250831 13:58:23.995819 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 130015us
I20250831 13:58:24.030886 20543 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:23.919628 21674 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:24.003482 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.036985 20643 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.038282 21459 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:24.016261 21585 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:24.044024 21770 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.046135 21599 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.046548 21732 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.000432 21508 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.005353 20466 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:23.984177 21562 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:23.921255 21659 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:23.995321 21544 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:23.994202 21554 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:24.000432 21731 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.014026 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.095587 20577 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.097678 21547 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:24.098369 21547 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.014190 20573 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:24.101011 21547 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.101702 21720 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:24.002460 21463 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:23.983122 21596 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.000942 21644 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:24.007063 21561 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:23.990739 21580 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:23.999963 21648 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20250831 13:58:24.123612 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.033236 20647 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:24.031164 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:24.035183 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.036127 20501 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:24.017609 21528 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.126163 21596 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.073010 21708 consensus_queue.cc:1035] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20250831 13:58:24.056447 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.127027 20576 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:24.102339 21720 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:24.128598 21812 log.cc:927] Time spent T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.339s user 0.003s sys 0.006s
I20250831 13:58:24.129753 21599 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.057269 20543 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:24.130508 21720 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:24.060158 21656 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.062481 21709 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3.
I20250831 13:58:24.024044 21588 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.131309 21375 consensus_queue.cc:260] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.132658 21375 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:24.133297 21588 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.133903 21588 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:24.134222 21588 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:24.023732 21737 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:24.134853 20574 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:24.134939 21824 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:24.026542 21745 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.135500 21824 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.135727 21745 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.066840 21765 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:24.137452 21824 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.067353 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 783) took 1097 ms. Trace:
I20250831 13:58:24.137946 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:24.052810 21618 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:24.138752 20645 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.018271 21349 raft_consensus.cc:695] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 2 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:24.139858 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.139987 21349 consensus_queue.cc:237] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:24.140595 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 785) took 1169 ms. Trace:
I20250831 13:58:24.141003 20647 rpcz_store.cc:276] 0831 13:58:22.970909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:23.008370 (+ 37461us) service_pool.cc:225] Handling call
0831 13:58:24.140444 (+1132074us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b540028b754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:24.140540 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2280754560}
I20250831 13:58:24.134546 21588 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.077785 21725 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:24.138796 20575 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.143518 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:58:24.131265 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 74103.8us
I20250831 13:58:24.072986 21508 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.076380 21544 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.146021 20543 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:24.137452 21745 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.075484 21732 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.147085 21732 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:24.106889 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.147722 21544 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.107625 20616 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:24.147215 21747 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:24.140632 20645 raft_consensus.cc:2391] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:24.149622 21747 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:24.110102 21804 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.155328 20573 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:24.110638 20570 raft_consensus.cc:1273] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:24.110961 21463 raft_consensus.cc:695] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:24.113641 21796 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:24.137912 20646 rpcz_store.cc:276] 0831 13:58:22.970199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.016707 (+1046508us) service_pool.cc:225] Handling call
0831 13:58:24.055023 (+ 38316us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.067002 (+ 11979us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:24.067033 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:24.067082 (+ 49us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:24.067114 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.067183 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:24.067287 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:24.115222 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 91854.5us
I20250831 13:58:24.009523 21556 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:24.117789 20569 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:24.118526 21783 consensus_queue.cc:260] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.115118 21735 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:24.077549 21554 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.049603 21727 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.175516 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.176116 20574 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:24.175988 21618 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:24.176610 21727 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.176959 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.177210 21727 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:24.177536 20574 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:24.123461 21580 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.142400 20647 raft_consensus.cc:1273] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:24.009615 21751 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.178884 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 784) took 1208 ms. Trace:
I20250831 13:58:24.179207 20647 rpcz_store.cc:276] 0831 13:58:22.970585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.141840 (+1171255us) service_pool.cc:225] Handling call
0831 13:58:24.142102 (+ 262us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.178630 (+ 36528us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.178836 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:24.180022 21751 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.180130 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:24.180749 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.181223 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.181738 20647 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.181770 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.025050 21503 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.184017 21503 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.184530 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.185194 20568 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:24.151611 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.186968 20645 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:24.187598 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 788) took 1215 ms. Trace:
I20250831 13:58:24.152786 20497 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
W20250831 13:58:24.153432 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 148172us
W20250831 13:58:24.189630 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:24.189971 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:24.155862 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 105344us
I20250831 13:58:24.173460 21735 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.190869 21738 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.191797 20572 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.192641 21735 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.192967 21731 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.194177 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.194780 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.197696 20617 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:24.198218 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:24.198940 21783 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:24.199031 20651 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:24.199338 21715 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250831 13:58:24.200184 20651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 493) took 2384 ms (client timeout 2346 ms). Trace:
W20250831 13:58:24.200528 20651 rpcz_store.cc:269] 0831 13:58:21.816094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:22.117004 (+300910us) service_pool.cc:225] Handling call
0831 13:58:24.200024 (+2083020us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b540029f254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:24.200126 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4449451264}
I20250831 13:58:24.201241 21735 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:24.201498 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:24.149425 21730 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:24.202857 20648 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:24.203286 21730 raft_consensus.cc:695] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:24.203760 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.204252 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 772) took 1897 ms. Trace:
I20250831 13:58:24.205605 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.204845 21583 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.206394 21583 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.205336 20648 rpcz_store.cc:276] 0831 13:58:22.306901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:22.359364 (+ 52463us) service_pool.cc:225] Handling call
0831 13:58:24.202533 (+1843169us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.203269 (+ 736us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.204087 (+ 818us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b540029f250. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:24.204204 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4238258432}
I20250831 13:58:24.169566 21804 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.207643 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.208158 20648 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.208714 21804 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.209954 21583 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.095012 21609 raft_consensus.cc:695] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:24.149418 20575 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:24.162552 20577 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:24.163714 20543 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:24.202123 20651 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
W20250831 13:58:24.215137 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 51506.8us
I20250831 13:58:24.215353 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.216498 20647 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:24.216797 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:24.167174 20541 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.131:45067 timed out after 2.346s (SENT) [suppressed 8 similar messages]
I20250831 13:58:24.218391 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.218991 20495 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:24.173377 21745 consensus_queue.cc:1046] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:24.174947 21554 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.176679 21561 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.177997 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.209563 21463 consensus_queue.cc:237] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.113631 21585 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.091961 20643 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:24.235951 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.217368 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:24.222256 20570 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.203306 20466 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:24.236915 20617 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:24.227747 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.236923 20647 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.237295 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.237843 20502 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:24.238193 20651 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:24.238801 20650 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:24.239039 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 793) took 1220 ms. Trace:
I20250831 13:58:24.239389 20647 rpcz_store.cc:276] 0831 13:58:23.019032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.238283 (+1219251us) service_pool.cc:225] Handling call
0831 13:58:24.238539 (+ 256us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.238745 (+ 206us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:24.238770 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:24.238811 (+ 41us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:24.238844 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.238896 (+ 52us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:24.238992 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:24.241165 20648 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:24.124467 21659 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.187902 20645 rpcz_store.cc:276] 0831 13:58:22.971849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.150027 (+1178178us) service_pool.cc:225] Handling call
0831 13:58:24.169107 (+ 19080us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.187363 (+ 18256us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.187541 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:24.242862 20466 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:24.242666 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 796) took 1165 ms. Trace:
I20250831 13:58:24.251827 20576 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:24.251243 20643 rpcz_store.cc:276] 0831 13:58:23.077608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.238894 (+1161286us) service_pool.cc:225] Handling call
0831 13:58:24.239176 (+ 282us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.239366 (+ 190us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:24.239387 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:24.239424 (+ 37us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:24.239449 (+ 25us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.239494 (+ 45us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:24.242615 (+ 3121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:24.231905 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.241494 20647 raft_consensus.cc:1273] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:24.206497 21528 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.241997 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.255580 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.256186 20577 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:24.242868 20617 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:24.235698 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.249055 20501 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 5. Ops: []
I20250831 13:58:24.257346 20568 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.211246 21708 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20250831 13:58:24.217257 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:24.255584 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 797) took 1177 ms. Trace:
I20250831 13:58:24.260954 20647 rpcz_store.cc:276] 0831 13:58:23.077926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.240184 (+1162258us) service_pool.cc:225] Handling call
0831 13:58:24.240417 (+ 233us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.255344 (+ 14927us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.255524 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:24.246754 20574 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.262133 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:24.262837 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 800) took 1126 ms. Trace:
I20250831 13:58:24.247610 21715 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.247792 21735 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:24.213512 20497 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:24.204200 21730 consensus_queue.cc:237] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.248698 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:24.253225 21745 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:24.265996 20571 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 5. Ops: []
I20250831 13:58:24.267613 20617 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.263312 20647 rpcz_store.cc:276] 0831 13:58:23.136352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.261872 (+1125520us) service_pool.cc:225] Handling call
0831 13:58:24.262205 (+ 333us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.262428 (+ 223us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:24.262465 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:24.262523 (+ 58us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:24.262568 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.262636 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:24.262773 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:24.297683 20647 raft_consensus.cc:1273] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250831 13:58:24.297703 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 54946.8us
I20250831 13:58:24.223858 21555 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:24.259469 21508 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:24.294939 20398 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54880 (request call id 34) took 1850 ms. Trace:
I20250831 13:58:24.210412 21796 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:24.253767 21565 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.264328 21738 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.268806 21544 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:24.302325 20647 raft_consensus.cc:1238] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:24.306455 20467 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.306963 21725 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.307706 21710 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.307973 21585 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.308908 21599 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:24.301043 21747 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.309512 20466 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.298828 20642 raft_consensus.cc:2391] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:24.267971 21715 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:24.300362 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:24.270754 21745 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:24.326499 21659 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.300686 20398 rpcz_store.cc:276] 0831 13:58:22.419424 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:22.419618 (+ 194us) service_pool.cc:225] Handling call
0831 13:58:24.269728 (+1850110us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:23.271429 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:23.436777 (+165348us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:23.436810 (+ 33us) write_op.cc:435] Acquired schema lock
0831 13:58:23.436827 (+ 17us) tablet.cc:660] Decoding operations
0831 13:58:23.467983 (+ 31156us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:23.468059 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:23.468082 (+ 23us) tablet.cc:683] Acquiring locks for 4 operations
0831 13:58:23.468471 (+ 389us) tablet.cc:699] Row locks acquired
0831 13:58:23.468490 (+ 19us) write_op.cc:260] PREPARE: finished
0831 13:58:23.468628 (+ 138us) write_op.cc:270] Start()
0831 13:58:23.468732 (+ 104us) write_op.cc:276] Timestamp: P: 1756648703468587 usec, L: 0
0831 13:58:23.468750 (+ 18us) op_driver.cc:348] REPLICATION: starting
0831 13:58:23.469317 (+ 567us) log.cc:844] Serialized 1761 byte log entry
0831 13:58:23.868978 (+399661us) op_driver.cc:464] REPLICATION: finished
0831 13:58:24.009961 (+140983us) write_op.cc:301] APPLY: starting
0831 13:58:24.011390 (+ 1429us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:24.017415 (+ 6025us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:24.051979 (+ 34564us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:24.071696 (+ 19717us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:24.071922 (+ 226us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0831 13:58:24.071999 (+ 77us) write_op.cc:312] APPLY: finished
0831 13:58:24.072304 (+ 305us) log.cc:844] Serialized 40 byte log entry
0831 13:58:24.072469 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:24.072603 (+ 134us) write_op.cc:454] Released schema lock
0831 13:58:24.072679 (+ 76us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:24.073100 (+ 421us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:24.073146 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":124254,"apply.run_cpu_time_us":3372,"apply.run_wall_time_us":68558,"num_ops":4,"prepare.queue_time_us":296921,"prepare.run_cpu_time_us":39499,"prepare.run_wall_time_us":377629,"raft.queue_time_us":130088,"raft.run_cpu_time_us":29885,"raft.run_wall_time_us":116169,"replication_time_us":400094,"thread_start_us":585659,"threads_started":4,"wal-append.queue_time_us":242767}]]}
I20250831 13:58:24.309271 21709 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:24.296409 20646 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:24.320936 20466 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:24.311439 21583 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:24.310133 21503 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:24.323963 20499 raft_consensus.cc:2391] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:24.315644 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:58:24.324708 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 798) took 1189 ms. Trace:
I20250831 13:58:24.326804 21561 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.324237 21747 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.327605 20644 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.317808 21725 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.328150 21731 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.328687 21556 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.329233 21715 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.320490 21710 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.352609 21599 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:24.365633 21599 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:24.366027 21599 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:24.366760 21710 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.352873 20636 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c14599239fd4c99b3331692a513a092 with delete_type TABLET_DATA_TOMBSTONED (TS 44d163094add49049cc749547d891602 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57684
I20250831 13:58:24.362857 20649 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.368824 21747 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.369789 20649 raft_consensus.cc:1238] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:24.357002 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.333906 21766 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:24.370618 20494 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:24.339488 21528 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.291138 21609 consensus_queue.cc:237] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.373072 21725 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.373565 21503 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:24.341238 21804 raft_consensus.cc:3053] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:24.299770 20493 raft_consensus.cc:1273] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:24.334424 20568 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:24.346199 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 801) took 1209 ms. Trace:
I20250831 13:58:24.328622 21565 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.344743 21796 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.359441 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.359093 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:24.426419 20617 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.323324 21724 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 13:58:24.374598 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 53802.5us
I20250831 13:58:24.375537 20643 rpcz_store.cc:276] 0831 13:58:23.135012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.253796 (+1118784us) service_pool.cc:225] Handling call
0831 13:58:24.254070 (+ 274us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.291722 (+ 37652us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:24.291752 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:24.291790 (+ 38us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:24.291817 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.324466 (+ 32649us) spinlock_profiling.cc:243] Waited 37.4 ms on lock 0x7b540006ac54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:24.324517 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:24.324640 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":86082816}
I20250831 13:58:24.378432 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:24.431450 20650 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:24.376123 20569 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:24.374402 21745 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:24.311882 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 144750us
I20250831 13:58:24.413138 20648 rpcz_store.cc:276] 0831 13:58:23.137090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.271495 (+1134405us) service_pool.cc:225] Handling call
0831 13:58:24.299439 (+ 27944us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.311278 (+ 11839us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:24.311298 (+ 20us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:24.311330 (+ 32us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:24.311348 (+ 18us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:24.345942 (+ 34594us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b54000307d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:24.345995 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:24.346135 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26854144}
I20250831 13:58:24.350055 20570 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:24.338439 21582 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.384210 21804 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:24.428568 20617 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:58:24.426703 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 67327.4us
I20250831 13:58:24.387893 20651 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.361635 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:24.434192 20466 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.384388 21503 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:24.423055 21783 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:24.445940 20616 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:24.360672 21562 raft_consensus.cc:1062] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: attempting to promote NON_VOTER ab7ade51fa134a1b814fab748f01d43d to VOTER
I20250831 13:58:24.374147 21375 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:24.373989 21583 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:24.419068 21603 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:24.439253 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.447801 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.431612 21561 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:24.334574 21588 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:24.447999 20541 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:24.448588 20467 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.429597 21724 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:24.442785 21715 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:24.463065 20650 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:24.456650 21603 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:24.450887 20646 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:24.452852 21205 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.463523 20398 catalog_manager.cc:5582] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 3 to 5, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 5 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } } }
I20250831 13:58:24.450111 21501 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:24.459919 20496 raft_consensus.cc:2391] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:24.468003 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.457762 21583 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:24.459004 21544 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:24.471791 20569 raft_consensus.cc:2366] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5.
I20250831 13:58:24.472728 20467 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.464715 21528 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:24.466743 21349 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:24.466929 21715 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:24.469275 21561 raft_consensus.cc:695] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:24.551677 21561 consensus_queue.cc:237] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.552611 21599 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.474879 20617 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:24.465564 21724 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:24.538283 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 61982.4us
I20250831 13:58:24.452984 21558 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20250831 13:58:24.470510 21725 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:24.459791 21827 consensus_queue.cc:1046] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20250831 13:58:24.455425 21783 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:24.539026 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 66319.5us
I20250831 13:58:24.466542 21603 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.547374 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.540531 21565 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:24.459496 21375 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.475679 21824 consensus_queue.cc:1046] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
W20250831 13:58:24.610299 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 135488us
I20250831 13:58:24.761430 20617 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:24.537963 21463 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.571159 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:24.570487 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.760512 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.761611 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.767776 20650 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.767967 20568 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:24.775568 20570 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:24.536032 21501 raft_consensus.cc:695] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:24.763419 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.764811 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:24.611346 21840 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250831 13:58:24.774266 20648 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:24.780822 20575 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.554845 21715 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:24.611728 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.554585 21599 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.782208 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.727552 21349 raft_consensus.cc:695] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 2 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:24.473038 21585 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.295s
I20250831 13:58:24.590162 21829 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.782339 20541 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:24.802273 20541 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:24.803510 20499 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:24.805251 20572 raft_consensus.cc:1238] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:24.805670 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.557657 21556 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:24.777056 21463 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.636713 21205 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:24.556499 21528 raft_consensus.cc:695] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:24.787140 21562 consensus_queue.cc:237] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 3, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:24.788553 20650 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:24.790791 21618 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:24.791405 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.821516 20574 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 4
I20250831 13:58:24.792093 21207 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:24.790426 21501 consensus_queue.cc:237] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.833953 20574 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:24.835402 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.836247 20574 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:24.838354 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.794050 20644 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.718559 21588 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:24.794227 20571 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.727749 21774 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:24.758324 21582 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.612015 21508 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:24.858085 20651 raft_consensus.cc:2391] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:24.766171 21656 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.769022 21580 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.807619 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.474274 21804 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.809641 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.812994 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:24.814518 20616 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:24.815256 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:24.866832 20572 raft_consensus.cc:2391] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:24.867261 20648 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.779296 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:24.828143 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.812234 21829 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.876422 20577 raft_consensus.cc:2391] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:24.844336 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.816246 21737 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:24.819059 21554 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20250831 13:58:24.862529 21508 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:24.866823 20570 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.808744 21473 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20250831 13:58:24.851778 21765 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.874982 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:24.877902 20644 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:24.806797 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:24.859228 21561 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.839964 21556 raft_consensus.cc:695] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:24.867116 21175 leader_election.cc:400] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5.
I20250831 13:58:24.886035 20648 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:24.875692 21588 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:24.865959 21656 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.902206 20571 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:24.880019 21804 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.879470 21210 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.904021 20570 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:24.879020 21737 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:24.891021 21175 leader_election.cc:403] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:24.889673 21508 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:24.893882 21770 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:24.866065 21774 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:24.920611 21375 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:24.905819 21555 raft_consensus.cc:2762] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
W20250831 13:58:24.921960 21839 log.cc:927] Time spent T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.169s user 0.001s sys 0.000s
I20250831 13:58:24.924257 21829 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:24.924973 21731 raft_consensus.cc:3053] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:24.925534 21731 raft_consensus.cc:738] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:24.925072 21556 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:24.893561 21561 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.932567 21561 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:24.943027 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:24.888046 21765 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.912072 21737 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:24.886176 21349 consensus_queue.cc:237] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:24.906519 21588 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.018872 21561 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:24.934012 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 112734us
I20250831 13:58:24.952147 21770 raft_consensus.cc:695] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:24.904937 21528 consensus_queue.cc:237] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:25.033597 20647 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.064s sys 0.177s
I20250831 13:58:24.926368 21731 consensus_queue.cc:260] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:25.043692 20647 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s user 0.064s sys 0.177s
I20250831 13:58:25.003233 21747 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:25.035579 21561 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.032083 21831 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:24.958086 21503 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:25.054267 21609 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.023s sys 0.043s
W20250831 13:58:25.074152 21609 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.024s sys 0.043s
W20250831 13:58:25.078009 21583 consensus_meta.cc:287] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.542s user 0.006s sys 0.000s
I20250831 13:58:25.078640 21770 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:25.051321 21774 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:25.049863 21463 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:25.097280 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 286728us
I20250831 13:58:25.066090 21747 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.100562 21463 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:25.097007 21565 raft_consensus.cc:695] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:25.071981 21831 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.101557 21603 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.102463 21656 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.078408 21583 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.143178 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:25.144057 21209 raft_consensus.cc:1811] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Handling vote request from an unknown peer 44d163094add49049cc749547d891602
I20250831 13:58:25.144793 21209 raft_consensus.cc:2366] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:25.121675 21463 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:25.239748 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:25.241156 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:25.254729 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.257175 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.259163 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.162492 21583 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.194110 21724 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.299029 20498 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:25.300632 20499 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:25.301069 21463 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.301617 21724 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.185518 21725 raft_consensus.cc:695] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:25.304299 21725 consensus_queue.cc:237] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.305392 20499 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
W20250831 13:58:25.186419 21745 consensus_meta.cc:287] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.742s user 0.011s sys 0.004s
I20250831 13:58:25.153786 21765 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.312013 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.188792 21831 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.201593 21863 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:25.313356 20502 raft_consensus.cc:2366] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4.
I20250831 13:58:25.313678 21745 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
W20250831 13:58:25.268134 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 166748us
W20250831 13:58:25.267019 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 445227us
I20250831 13:58:25.315037 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.316838 21745 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.271625 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:25.318274 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:25.273049 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 810) took 1702 ms. Trace:
I20250831 13:58:25.291399 21503 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.172101 21588 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.281168 21738 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:25.320196 21503 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:25.320432 21738 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.318857 20647 rpcz_store.cc:276] 0831 13:58:23.570078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.317104 (+747026us) service_pool.cc:225] Handling call
0831 13:58:24.342168 (+ 25064us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:24.343788 (+ 1620us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:24.343814 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:24.419848 (+ 76034us) log.cc:844] Serialized 26 byte log entry
0831 13:58:25.044076 (+624228us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:25.085973 (+ 41897us) log.cc:844] Serialized 14 byte log entry
0831 13:58:25.086788 (+ 815us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:25.086858 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:25.272400 (+185542us) raft_consensus.cc:1683] finished
0831 13:58:25.272447 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:25.272778 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":42052,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":1552,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":225,"spinlock_wait_cycles":9984,"thread_start_us":592015,"threads_started":1,"wal-append.queue_time_us":713957}
I20250831 13:58:25.321015 21503 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.281426 21207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47742 (request call id 53) took 1213 ms. Trace:
I20250831 13:58:25.282160 21824 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:25.322553 20646 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:25.302407 21724 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:25.322829 21207 rpcz_store.cc:276] 0831 13:58:24.068296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.117209 (+ 48913us) service_pool.cc:225] Handling call
0831 13:58:25.280922 (+1163713us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54005e1ad4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:25.281165 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":252978304}
I20250831 13:58:25.323115 21824 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.324002 21724 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.325713 21824 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.283733 21853 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:25.327060 21853 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.271669 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.290419 21561 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.290025 21565 consensus_queue.cc:237] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.291368 21603 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.293051 21585 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:25.313975 21863 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:25.194098 21737 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.335469 21503 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.334095 20642 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:25.336942 21737 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.336139 21853 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.318301 20617 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.319586 20541 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:25.338578 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.347034 21585 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.350266 21585 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.351785 21609 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:25.379679 21783 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.379778 21609 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.358664 21863 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.360776 21738 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.364728 20616 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:25.366433 20499 raft_consensus.cc:2366] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:25.368064 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.369657 21599 raft_consensus.cc:3053] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:25.375480 21596 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:25.376255 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.378544 21724 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:25.393747 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 74516.3us
I20250831 13:58:25.400014 21596 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.400328 21738 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
W20250831 13:58:25.400290 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 82202.8us
I20250831 13:58:25.401146 21738 raft_consensus.cc:695] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:25.401338 21770 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:25.401407 20617 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.401878 21770 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:25.402796 21751 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:25.402734 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.403326 21751 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:25.403750 21751 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:25.405544 20642 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:25.402480 21738 consensus_queue.cc:237] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.388298 21599 raft_consensus.cc:738] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:25.406508 21731 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:25.427492 21751 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.443580 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:25.432008 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:25.419011 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.448887 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.449218 20575 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:25.449362 21747 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.432363 20500 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:25.449952 20570 raft_consensus.cc:2408] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:25.450177 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.423122 20617 leader_election.cc:400] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173) with higher term. Message: Invalid argument: T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:25.451152 20617 leader_election.cc:403] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:25.450992 21599 consensus_queue.cc:260] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.412427 21609 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.446033 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.429046 21841 tablet_replica.cc:331] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: stopping tablet replica
I20250831 13:58:25.452128 21855 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.402288 21596 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.452517 20568 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
W20250831 13:58:25.410171 21852 log.cc:927] Time spent T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.251s user 0.002s sys 0.000s
W20250831 13:58:25.426327 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 957970us
I20250831 13:58:25.437592 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 541) took 1680 ms. Trace:
I20250831 13:58:25.402376 21770 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:25.418164 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 799) took 2282 ms. Trace:
I20250831 13:58:25.447099 20502 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:25.448892 20494 raft_consensus.cc:2366] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3.
I20250831 13:58:25.407125 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.411815 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.455611 20466 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:25.480000 20495 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:25.450999 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.437439 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:57684 (request call id 187) took 2482 ms. Trace:
I20250831 13:58:25.482760 20500 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:25.486246 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.460268 21599 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.482615 20643 rpcz_store.cc:276] 0831 13:58:22.955041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.435333 (+1480292us) service_pool.cc:225] Handling call
0831 13:58:25.422508 (+987175us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b54002ea754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8facd0 00007fb9bc97eeea 00007fb9c18d760e 00007fb9baf74cc7 00007fb9baf74c52 00007fb9baf74b81 00007fb9baf74afc 00007fb9baf73db2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:25.437210 (+ 14702us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2263881984}
I20250831 13:58:25.513435 20498 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:25.430918 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.482196 21855 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173) with higher term. Message: Invalid argument: T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:25.515209 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.407791 21174 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:25.515743 20541 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:25.516837 20499 raft_consensus.cc:2366] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:25.517127 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.518548 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.519239 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:25.519610 21508 consensus_meta.cc:287] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.569s user 0.009s sys 0.005s
I20250831 13:58:25.526381 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.528367 20495 raft_consensus.cc:2391] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:25.541563 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:25.551659 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 151925us
I20250831 13:58:25.558954 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.563529 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:25.568638 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 161085us
I20250831 13:58:25.569358 21863 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:25.455052 20649 rpcz_store.cc:276] 0831 13:58:23.756879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.370495 (+613616us) service_pool.cc:225] Handling call
0831 13:58:24.375826 (+ 5331us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:24.794918 (+419092us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:24.795123 (+ 205us) log.cc:844] Serialized 14 byte log entry
0831 13:58:25.040981 (+245858us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:25.041046 (+ 65us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:25.041101 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:25.437105 (+396004us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b540015bdd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:25.437174 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:25.437394 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":23860,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":5116,"raft.queue_time_us":100,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5776,"spinlock_wait_cycles":963387264,"thread_start_us":245454,"threads_started":1,"wal-append.queue_time_us":353776}
I20250831 13:58:25.575019 20502 raft_consensus.cc:2391] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:25.570637 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 55694.3us
W20250831 13:58:25.575901 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 211429us
I20250831 13:58:25.547483 21618 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250831 13:58:25.577395 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 47848.9us
I20250831 13:58:25.575289 21609 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:25.604671 20494 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:25.605695 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.610196 20617 leader_election.cc:400] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:25.610541 20617 leader_election.cc:403] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20250831 13:58:25.610903 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:25.611846 20576 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:25.589111 21863 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:25.576726 20493 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:25.615319 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:25.613662 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 158182us
I20250831 13:58:25.595223 21747 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:25.620424 21554 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.620573 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:25.621510 20575 raft_consensus.cc:2391] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:25.622206 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:25.623189 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.616245 20499 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:25.616034 21863 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:25.619628 21609 raft_consensus.cc:695] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:25.588492 21508 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.636503 20570 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.636054 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:25.641510 21580 consensus_meta.cc:287] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.743s user 0.005s sys 0.013s
I20250831 13:58:25.641875 20569 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:25.642040 21580 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:25.642283 21349 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:25.641865 20500 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:25.643098 21349 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:25.644117 20498 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:25.644385 20500 raft_consensus.cc:1273] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:25.650892 21747 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:25.651513 21747 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:25.651696 21473 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:25.652457 21473 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.633982 21751 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.654435 21473 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.665472 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.667943 20568 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:25.675204 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.676100 20569 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:25.679450 21205 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 13:58:25.681496 20543 leader_election.cc:400] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4.
I20250831 13:58:25.681929 20543 leader_election.cc:403] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:25.683806 21528 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4.
I20250831 13:58:25.685771 20543 leader_election.cc:400] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:25.678548 21826 consensus_queue.cc:1046] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
W20250831 13:58:25.690539 21210 consensus_meta.cc:287] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.772s user 0.013s sys 0.005s
I20250831 13:58:25.690951 21210 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:25.691092 21826 raft_consensus.cc:3053] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:25.691547 21826 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:25.471307 20645 rpcz_store.cc:276] 0831 13:58:23.135809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.254632 (+1118823us) service_pool.cc:225] Handling call
0831 13:58:24.259296 (+ 4664us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:24.268436 (+ 9140us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:24.275733 (+ 7297us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:24.319955 (+ 44222us) log.cc:844] Serialized 26 byte log entry
0831 13:58:24.456438 (+136483us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:25.416955 (+960517us) spinlock_profiling.cc:243] Waited 47.5 ms on lock 0x7b180009b7a0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9bc5c255c 00007fb9b99f777b 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a11805 00007fb9c035b327 00007fb9bc980957 00007fb9bc99006f
0831 13:58:25.417133 (+ 178us) log.cc:844] Serialized 14 byte log entry
0831 13:58:25.417535 (+ 402us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:25.417659 (+ 124us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:25.417730 (+ 71us) raft_consensus.cc:1683] finished
0831 13:58:25.417826 (+ 96us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:25.417984 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":947,"prepare.queue_time_us":174486,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":7737,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":3925,"spinlock_wait_cycles":109276928,"thread_start_us":286612,"threads_started":2,"wal-append.queue_time_us":142993}
I20250831 13:58:25.692265 21796 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5.
I20250831 13:58:25.692571 21826 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.693914 21826 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:25.696785 21715 consensus_meta.cc:287] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.885s user 0.002s sys 0.016s
I20250831 13:58:25.697255 21715 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.702663 20570 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:25.661993 21349 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:25.688033 21871 log.cc:927] Time spent T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
W20250831 13:58:25.712064 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 45645.4us
W20250831 13:58:25.711997 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 52197us
I20250831 13:58:25.714398 20617 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.641983 21858 consensus_queue.cc:1046] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20250831 13:58:25.717324 20541 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:25.717553 20617 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:25.721423 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:25.723608 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:25.724277 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:25.725519 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 85760.5us
I20250831 13:58:25.729597 20575 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:25.729956 21737 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:25.730453 21737 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:25.730484 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.730983 21737 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.731170 20499 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:25.732092 21876 log.cc:927] Time spent T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250831 13:58:25.733942 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.736294 20543 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:25.739145 21751 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:25.740270 21708 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:25.740536 21556 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:25.734898 21580 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:25.741191 21556 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:25.741612 21580 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:25.741750 21585 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:25.742218 21580 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.742391 21585 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:25.743295 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.744216 20498 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:25.744745 21585 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:25.740465 21751 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:25.741468 21708 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.748509 20541 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:25.747690 20616 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:25.725404 21715 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.756781 21585 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:25.758013 21463 raft_consensus.cc:3053] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:25.760036 21708 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.755055 21751 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:25.788499 21585 raft_consensus.cc:695] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:25.794145 21585 consensus_queue.cc:237] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:25.795254 21863 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:25.798103 21863 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:25.742359 21872 log.cc:927] Time spent T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.137s user 0.001s sys 0.000s
I20250831 13:58:25.788982 21588 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:25.754364 21855 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20250831 13:58:25.799180 21588 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:25.799566 21588 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.741677 21556 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.800284 21855 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:25.800853 21855 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.801007 21737 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.802836 21855 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.803567 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:25.804422 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 66339.3us
I20250831 13:58:25.805578 21855 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:25.805774 21580 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.762005 20466 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:25.789075 21555 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:25.807296 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 45476.8us
I20250831 13:58:25.807806 21555 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.808463 21865 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:25.808966 21865 raft_consensus.cc:695] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:25.809695 21555 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.810252 21555 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:25.819197 21737 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.780272 21554 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:25.717986 21609 consensus_queue.cc:237] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.821606 20649 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:25.781539 21508 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.747248 21473 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:25.822937 21473 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.823726 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:25.824482 20649 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:25.824972 21473 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:25.782742 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 49099.7us
I20250831 13:58:25.774778 21747 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.826037 21588 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.826875 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:25.827492 21205 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:25.827934 21588 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.772099 20467 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:25.788434 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 54144.7us
I20250831 13:58:25.788611 21731 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:25.829274 21731 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.787621 21463 raft_consensus.cc:738] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:25.831137 21731 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.831647 21463 consensus_queue.cc:260] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.833004 21463 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:25.834985 21827 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:25.835389 21827 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:25.835767 21580 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.789536 21210 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 LEARNER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20250831 13:58:25.836833 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 64862us
I20250831 13:58:25.837455 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.817234 21855 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:25.810045 21865 consensus_queue.cc:237] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.838109 20467 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.839061 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:25.839787 20646 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.840265 21503 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:25.840837 21503 raft_consensus.cc:695] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:25.818969 21555 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.819489 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.784060 21528 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:25.835246 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.835829 21827 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:25.860329 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.805124 20616 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:25.838056 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.841290 20467 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.716706 21349 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20250831 13:58:25.803628 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:25.833011 20493 raft_consensus.cc:1273] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:25.838060 20498 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:25.843631 20651 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:25.845985 20499 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:25.838374 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:58:25.859082 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:25.790776 21783 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
W20250831 13:58:25.867390 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 63844.1us
I20250831 13:58:25.874316 20467 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.775110 21770 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.869556 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.874392 21857 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:25.869688 20650 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:25.877913 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.878549 20651 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:25.883292 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 44988.9us
I20250831 13:58:25.885300 21856 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:25.885457 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:25.890928 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:25.895545 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:25.895642 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.896401 20499 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:25.894140 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.851737 21725 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250831 13:58:25.902568 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.905084 20617 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.875062 20642 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:25.889724 21731 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:25.913211 20570 raft_consensus.cc:2391] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:25.908776 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:25.914795 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 109740us
I20250831 13:58:25.916798 20651 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:25.871774 21855 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.915254 20643 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:25.874877 21528 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:25.899547 21856 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:25.909544 21599 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:25.914248 21473 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:25.914292 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.918991 21731 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:25.911489 21857 raft_consensus.cc:695] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:25.896157 21174 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:25.933859 20568 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:25.902458 21556 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:25.939132 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.976581 20643 raft_consensus.cc:1273] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:25.974258 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.980219 21863 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:25.980787 21863 raft_consensus.cc:695] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:25.981781 21863 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:25.942000 21731 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:25.987002 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:25.987764 20575 raft_consensus.cc:2391] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:25.989343 20499 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:25.990331 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:25.991118 20568 raft_consensus.cc:2366] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:25.991320 21857 consensus_queue.cc:237] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.992828 21556 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:25.935459 21858 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:25.992980 21855 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:25.993096 20571 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:25.925076 21618 consensus_queue.cc:1035] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20250831 13:58:25.939692 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:25.997387 20617 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:25.997607 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:25.977926 20644 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:26.011444 21645 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:26.012063 21645 raft_consensus.cc:695] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:26.012356 21855 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:25.975391 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.005535 20498 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:26.012879 21855 raft_consensus.cc:695] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:25.939538 21503 consensus_queue.cc:237] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.019120 21855 consensus_queue.cc:237] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.020236 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.013152 20616 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:25.997948 21856 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.026486 20576 raft_consensus.cc:1811] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Handling vote request from an unknown peer 44d163094add49049cc749547d891602
I20250831 13:58:26.027639 20576 raft_consensus.cc:2366] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:26.030045 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:26.006052 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 93914.2us
I20250831 13:58:26.031394 20617 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:25.996466 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 119122us
I20250831 13:58:26.007014 21747 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.013015 21645 consensus_queue.cc:237] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.037551 21841 raft_consensus.cc:2241] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250831 13:58:26.040622 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.045107 20644 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:25.972918 21473 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:25.973724 21751 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:26.059665 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 86591.9us
I20250831 13:58:26.070130 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:26.082873 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 65185.8us
I20250831 13:58:26.033433 21555 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.085542 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.087652 21783 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:26.087585 20616 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:26.043860 21508 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:26.090581 21841 raft_consensus.cc:2270] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250831 13:58:26.059665 21349 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:26.091498 21473 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.091946 20649 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:26.092070 21751 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.093725 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.089706 21770 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.099346 20647 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.097254 20398 catalog_manager.cc:5582] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 2, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 2 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:26.092360 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.106797 20573 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:26.096091 21783 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:26.112217 21783 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.111671 20398 catalog_manager.cc:5582] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 reported cstate change: term changed from 3 to 4, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 4 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:26.112881 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.112998 21841 ts_tablet_manager.cc:1905] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250831 13:58:26.113270 20398 catalog_manager.cc:5582] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:26.114100 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.115319 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.117373 20648 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.125033 20398 catalog_manager.cc:5582] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 2, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:26.128804 20398 catalog_manager.cc:5582] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 5, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 5 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
W20250831 13:58:26.130914 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 43494.2us
I20250831 13:58:26.131775 20616 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:26.134701 20646 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:26.135408 21583 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:26.136319 20651 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:26.108520 21349 raft_consensus.cc:695] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:26.138967 21528 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:26.138909 21554 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:26.146041 20648 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:26.146737 20650 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:26.064601 21856 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.050555 21858 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:26.149339 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.130426 20398 catalog_manager.cc:5582] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 2, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
W20250831 13:58:26.114881 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 84240.3us
I20250831 13:58:26.155972 20616 leader_election.cc:400] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:26.157687 21554 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250831 13:58:26.160643 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.159812 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:26.139210 21349 consensus_queue.cc:237] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.163237 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.163923 20646 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:26.164196 20398 catalog_manager.cc:5582] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:26.164749 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.166260 20541 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:26.168404 20616 leader_election.cc:400] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:26.169235 20541 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:26.169988 21554 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:26.170274 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:26.166184 20398 catalog_manager.cc:5582] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:26.167286 20643 raft_consensus.cc:1273] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:26.171104 20541 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:26.180361 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.181042 20541 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:26.170014 21473 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:26.179903 20398 catalog_manager.cc:5582] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:26.183274 21555 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:26.189333 20573 raft_consensus.cc:1273] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:26.189492 21555 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:26.189944 21555 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:26.184938 21528 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:26.195660 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.196197 21528 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:26.196429 20570 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:26.196647 21528 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:26.199518 20569 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250831 13:58:26.200428 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 165827us
I20250831 13:58:26.203624 20647 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:26.170454 21731 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:26.211133 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.211627 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.211961 20568 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:26.212684 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.213011 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:26.213541 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 120198us
I20250831 13:58:26.213670 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.214109 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.214875 20650 raft_consensus.cc:2391] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:26.213131 20648 raft_consensus.cc:1273] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:26.216969 20467 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:26.218878 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:26.219255 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.219658 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.220415 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.220697 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.221289 20575 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:26.221535 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.222162 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.222759 20571 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:26.224418 21856 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.222272 21208 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:26.205842 21588 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.225811 21588 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.227250 20398 catalog_manager.cc:5582] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:26.229138 20649 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:26.230059 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.185613 21585 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20250831 13:58:26.207561 21599 raft_consensus.cc:695] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:26.235231 21826 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.235184 21599 consensus_queue.cc:237] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.235884 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.237038 21896 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:26.237473 21896 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:26.237810 21896 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.228734 20398 catalog_manager.cc:5582] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:26.239897 20571 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:26.185840 21473 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:26.186012 21708 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:26.186645 21609 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:26.239058 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.245229 21473 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.248613 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.267938 20569 raft_consensus.cc:2391] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:26.221347 21827 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.280759 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.281646 20648 raft_consensus.cc:2391] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:26.281733 20643 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:26.282889 20494 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:26.283668 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.284235 20650 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:26.285966 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.312898 20643 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 13:58:26.280436 21827 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.328487 21765 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:26.331085 21767 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.250502 21708 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:26.250377 21609 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:26.235827 21826 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.337132 21765 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.340312 21767 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.348579 21708 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:26.363744 21609 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.391127 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:26.392982 20648 raft_consensus.cc:2391] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:26.393399 20570 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:58:26.397064 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:26.397625 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:26.401727 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.402295 20498 raft_consensus.cc:2366] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5.
I20250831 13:58:26.406028 20467 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:26.408941 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.382300 21826 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.410619 20650 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:26.408154 21585 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:26.412580 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.413271 20501 raft_consensus.cc:2366] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5.
I20250831 13:58:26.413856 21765 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.396519 21737 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:26.414323 20499 raft_consensus.cc:1238] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:26.412724 21585 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:26.416159 21580 consensus_queue.cc:1046] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:26.416428 21585 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.416936 21856 raft_consensus.cc:3053] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:26.417330 21856 raft_consensus.cc:738] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:26.418114 21856 consensus_queue.cc:260] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:26.419168 21856 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.399420 21827 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250831 13:58:26.420989 21737 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.404618 20646 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:26.403818 21528 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.383975 21555 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.432248 21826 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:26.435909 21826 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:26.438341 21826 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:26.432178 21765 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:26.426945 21737 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.439278 21765 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.441151 21765 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.442667 21767 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.450142 20647 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:26.450392 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.450703 21783 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.451118 20650 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:26.451192 21783 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.458237 20650 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:26.459388 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:26.460075 21596 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:26.460425 21596 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:26.460776 21596 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.460638 21508 consensus_queue.cc:1046] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:26.461558 21767 raft_consensus.cc:3053] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:26.461917 21767 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:26.462695 21767 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.463809 21767 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:26.467285 21905 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250831 13:58:26.463016 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.473831 20467 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:26.477239 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.455329 21588 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.495647 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.496670 21827 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:26.497347 21827 raft_consensus.cc:695] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:26.497640 21909 raft_consensus.cc:2762] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 13:58:26.498407 21827 consensus_queue.cc:237] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.500306 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:26.500905 21911 log.cc:927] Time spent T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20250831 13:58:26.501463 21783 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.498373 20502 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:26.494465 20467 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:26.506466 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.507970 20617 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:26.516556 20570 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:26.517174 20568 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20250831 13:58:26.515390 20571 raft_consensus.cc:1273] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:26.516480 21508 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:26.532331 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.528065 21508 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:26.534605 21508 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.534878 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.536334 21528 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.537586 20466 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.538522 21708 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.539135 20466 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.540503 21708 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.541527 20571 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:26.542626 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.543262 20570 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.518590 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:26.518759 21588 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:26.548507 21896 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.548930 21588 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.551836 21715 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.552493 20568 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:26.552300 21715 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.554152 21715 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.559558 21555 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.562712 20543 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:26.521602 21858 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:26.564414 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.564970 20577 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:26.550433 21896 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.511731 21907 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:26.574566 21907 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:26.574994 21907 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:26.574795 20543 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:26.558579 20570 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:26.568639 20568 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:26.578985 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 49523us
I20250831 13:58:26.585337 20494 raft_consensus.cc:1273] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250831 13:58:26.587416 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 48408.2us
I20250831 13:58:26.597491 20571 raft_consensus.cc:1273] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:26.599571 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.606992 21919 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:26.607589 21919 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.608496 21765 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.609010 21765 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.609728 21919 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.610220 21919 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:26.616707 20651 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 13:58:26.577280 21914 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:26.589933 21555 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:26.588744 21725 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:26.611104 21737 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:26.610895 21765 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.591892 21708 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20250831 13:58:26.594434 21855 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:26.654327 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:26.651928 20466 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.650261 21555 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:26.647889 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:26.664093 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.665083 21582 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:26.659513 21863 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20250831 13:58:26.666209 21731 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:26.667222 21916 log.cc:927] Time spent T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20250831 13:58:26.646888 21858 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.014s
I20250831 13:58:26.670457 21473 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.672175 20466 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.672462 21473 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.648597 21609 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.610634 21919 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.626395 21508 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:26.662837 21915 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20250831 13:58:26.657397 21827 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:26.685364 21588 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.685914 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.691190 20569 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:26.648450 21596 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.691972 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
W20250831 13:58:26.693106 21910 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.295s user 0.000s sys 0.003s
I20250831 13:58:26.693243 21596 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.658825 20651 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:26.693985 20651 raft_consensus.cc:604] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:26.694377 20651 raft_consensus.cc:612] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:58:26.657450 21855 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.688292 20498 raft_consensus.cc:1273] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:26.688634 20494 raft_consensus.cc:1273] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:26.688573 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.696655 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.662751 21555 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.673413 20642 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
W20250831 13:58:26.701036 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:26.703115 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.705835 20568 raft_consensus.cc:1238] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:26.696259 21855 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.707759 20466 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.708858 20466 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.709352 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.709865 20466 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.710121 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.710440 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.711163 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.711246 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.711861 21599 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:26.712227 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.712455 21599 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:26.711431 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.713146 21599 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:26.713241 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.709515 20495 raft_consensus.cc:1273] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250831 13:58:26.713567 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.713924 20495 pending_rounds.cc:85] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 1
I20250831 13:58:26.714216 20495 pending_rounds.cc:107] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 13:58:26.714545 20495 raft_consensus.cc:2887] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:26.708876 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.714226 20500 raft_consensus.cc:2366] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:26.715507 20501 raft_consensus.cc:2391] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:26.715939 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.716223 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.717267 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.717868 20501 raft_consensus.cc:2366] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:26.718963 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.715934 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.719663 20574 raft_consensus.cc:2391] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:26.720306 21896 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:26.721416 20568 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250831 13:58:26.722306 20574 raft_consensus.cc:1273] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:26.721417 20466 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.712375 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.723873 20543 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:26.714339 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.739652 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.741981 20574 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:26.724097 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.718536 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.724570 21896 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:26.728377 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.733068 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.741119 21865 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:58:26.741278 21923 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:26.745854 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:26.748436 20467 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.748786 20543 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:26.749647 21473 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:26.750346 21473 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:26.751127 21562 consensus_queue.cc:1035] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:26.751684 21565 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:26.748822 20574 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:26.752501 21562 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:26.747663 20500 raft_consensus.cc:2374] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3. Re-sending same reply.
I20250831 13:58:26.753377 21565 consensus_queue.cc:1046] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:26.754424 21565 raft_consensus.cc:3053] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:26.754516 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:26.755090 21565 raft_consensus.cc:738] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:26.756218 21588 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:26.744755 20499 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250831 13:58:26.757126 21556 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:26.746165 20466 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:26.756867 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:26.757083 21565 consensus_queue.cc:260] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.759020 21556 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:26.755870 21896 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:26.744947 20501 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:26.759393 21565 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.760213 21890 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:26.760984 21890 raft_consensus.cc:695] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:26.760213 21556 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:26.761932 21890 consensus_queue.cc:237] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.763510 20574 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:26.763773 21599 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:26.654949 21737 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.768227 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.742659 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:26.707216 21715 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:26.752079 21865 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:26.770099 20543 leader_election.cc:400] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:26.791980 20543 leader_election.cc:403] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250831 13:58:26.791925 20466 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:26.812691 20577 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:26.779382 20647 raft_consensus.cc:2408] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:26.814071 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:26.815246 20568 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:26.789680 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.769176 20498 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:26.820593 20569 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:26.817023 20570 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:26.828030 20543 leader_election.cc:400] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:26.828450 20543 leader_election.cc:403] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20250831 13:58:26.775703 21922 log.cc:927] Time spent T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20250831 13:58:26.846453 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.847098 20500 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:26.821828 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 169160us
I20250831 13:58:26.775063 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:26.777269 21609 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:26.776337 21528 raft_consensus.cc:3053] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:26.780803 21205 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250831 13:58:26.852488 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:26.791298 21896 raft_consensus.cc:695] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:26.773602 21508 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:26.816092 21588 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:26.770514 21599 consensus_queue.cc:1035] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.084s
I20250831 13:58:26.774839 21841 ts_tablet_manager.cc:1918] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250831 13:58:26.808327 21580 consensus_queue.cc:1035] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20250831 13:58:26.851362 21826 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:26.864419 20543 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:26.831239 21731 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20250831 13:58:26.873114 20645 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:26.873658 20500 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
W20250831 13:58:26.871052 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 79221us
I20250831 13:58:26.881477 20645 raft_consensus.cc:604] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:26.881932 20645 raft_consensus.cc:612] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:58:26.856928 21528 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:26.884353 20498 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:26.886253 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.886922 20617 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:26.870383 21921 consensus_queue.cc:1046] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20250831 13:58:26.881129 21609 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:26.899835 21751 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.900257 21783 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:26.901506 21827 raft_consensus.cc:3053] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:26.901641 21659 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:26.903225 21176 leader_election.cc:400] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5.
I20250831 13:58:26.904058 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:26.907750 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.936972 20647 raft_consensus.cc:2391] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:26.908825 20651 raft_consensus.cc:1273] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:26.909404 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:26.938671 20648 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:26.889547 21890 raft_consensus.cc:2762] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20250831 13:58:26.929706 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.942984 20497 raft_consensus.cc:2391] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:26.861905 21909 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20250831 13:58:26.887863 21923 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20250831 13:58:26.894610 21841 log.cc:1199] T 0c14599239fd4c99b3331692a513a092 P 44d163094add49049cc749547d891602: Deleting WAL directory at /tmp/dist-test-taskm8HSX8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1756648613964447-19366-0/minicluster-data/ts-2-root/wals/0c14599239fd4c99b3331692a513a092
I20250831 13:58:26.883432 21588 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20250831 13:58:26.942941 20576 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:26.950301 20645 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:26.930243 21865 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:26.926308 21708 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:26.958530 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:26.947695 21659 raft_consensus.cc:695] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:26.960417 20651 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50faaafc366a4da79cca52e16429da72"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:26.983979 20651 raft_consensus.cc:604] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:26.984429 20651 raft_consensus.cc:612] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250831 13:58:26.961182 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 58209.3us
I20250831 13:58:26.942137 21896 consensus_queue.cc:237] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:26.965801 21737 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:26.966881 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 108717us
I20250831 13:58:26.934216 21827 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:26.927170 21907 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:26.935293 21751 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.035977 20648 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:27.044680 20467 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:26.971391 21919 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:27.051915 20543 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:27.036269 21910 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250831 13:58:27.057092 20576 raft_consensus.cc:1273] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:27.036059 21737 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:27.076275 20648 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "ab7ade51fa134a1b814fab748f01d43d"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:27.077337 20648 raft_consensus.cc:604] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Received request to transfer leadership to TS ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:27.077764 20648 raft_consensus.cc:612] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:58:26.985384 21708 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.015041 21765 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.067s
I20250831 13:58:27.059758 21890 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
W20250831 13:58:27.079070 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 148223us
I20250831 13:58:26.956970 21528 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.074560 21919 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:27.049219 21857 consensus_queue.cc:1046] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20250831 13:58:27.103912 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:27.104141 20466 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:27.116573 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:27.117300 20497 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:27.118992 20498 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:27.095774 21919 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:27.080768 21737 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.135622 20648 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:27.136396 20648 raft_consensus.cc:604] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:27.147035 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:27.090049 21827 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.154337 21708 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:27.156083 21932 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:27.156750 21932 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:27.161332 21926 log.cc:927] Time spent T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.214s user 0.000s sys 0.001s
I20250831 13:58:27.104516 21659 consensus_queue.cc:237] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.167879 20576 raft_consensus.cc:1273] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:27.140000 21919 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:27.169816 21890 raft_consensus.cc:695] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:27.174356 21907 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:27.174559 21929 log.cc:927] Time spent T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20250831 13:58:27.172936 21827 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:27.183996 21857 raft_consensus.cc:3053] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:27.196864 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:27.203263 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:27.198350 21857 raft_consensus.cc:738] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:27.193902 21890 consensus_queue.cc:237] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:27.205101 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:27.205035 21857 consensus_queue.cc:260] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.206357 21857 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:27.218778 20616 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:27.198982 21708 consensus_queue.cc:1046] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20250831 13:58:27.208366 21841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:57684 (request call id 188) took 2896 ms. Trace:
W20250831 13:58:27.229969 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 125936us
I20250831 13:58:27.202178 21751 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:27.235033 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:27.235322 20645 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:27.236195 20616 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:27.202720 21932 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:27.220494 21599 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:27.275740 20616 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:27.278604 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.279114 21715 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:27.279670 21715 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:27.279879 21943 log.cc:927] Time spent T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20250831 13:58:27.280066 21715 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:27.281599 21942 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250831 13:58:27.281736 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:27.282152 21596 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:58:27.282977 21936 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250831 13:58:27.233997 21841 rpcz_store.cc:276] 0831 13:58:24.343229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:24.352556 (+ 9327us) service_pool.cc:225] Handling call
0831 13:58:24.700412 (+347856us) ts_tablet_manager.cc:1145] Deleting tablet 0c14599239fd4c99b3331692a513a092
0831 13:58:24.719555 (+ 19143us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0831 13:58:25.428594 (+709039us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b5400030054. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97efa9 00007fb9c1923282 00007fb9c1953260 00007fb9c193a63d 00007fb9c193a5e9 00007fb9c193a579 00007fb9c193a541 00007fb9c193976d 0000000000379254 0000000000379099
0831 13:58:26.226048 (+797454us) tablet_metadata.cc:641] Metadata flushed
0831 13:58:26.774435 (+548387us) tablet_metadata.cc:641] Metadata flushed
0831 13:58:27.197730 (+423295us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":579314,"spinlock_wait_cycles":605949184,"tablet-delete.queue_time_us":223510,"thread_start_us":84939,"threads_started":1}
W20250831 13:58:27.290412 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 59420.8us
I20250831 13:58:27.291534 20466 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:27.293663 21827 consensus_queue.cc:1035] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:27.295464 21952 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:27.295707 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:27.239926 21580 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:27.291553 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:27.300340 20570 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
W20250831 13:58:27.296185 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 106518us
I20250831 13:58:27.278371 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:27.304324 20385 catalog_manager.cc:4928] TS 44d163094add49049cc749547d891602 (127.18.233.131:45067): tablet 0c14599239fd4c99b3331692a513a092 (table test-workload [id=bf9639b2cb72453d8b67d48646dc2938]) successfully deleted
I20250831 13:58:27.225553 21508 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20250831 13:58:27.222003 21583 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20250831 13:58:27.276648 21855 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:27.307997 20616 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:27.315097 20570 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:27.308133 21858 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:27.328028 20649 raft_consensus.cc:1273] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:27.302814 21599 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
W20250831 13:58:27.348731 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44758.4us
I20250831 13:58:27.322755 21737 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:27.367897 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.369416 20645 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
W20250831 13:58:27.345598 21555 consensus_meta.cc:287] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.573s user 0.010s sys 0.001s
W20250831 13:58:27.500883 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 322041us
I20250831 13:58:27.363682 21855 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:27.374154 21609 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:27.484678 21751 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:27.483889 21783 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:27.573473 21659 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:27.493721 21555 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.553033 21858 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:27.574862 21659 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:27.582722 21951 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.256s user 0.001s sys 0.000s
I20250831 13:58:27.624509 21659 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:27.583585 20570 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:27.624959 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:27.564025 21508 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:27.583657 21715 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.586587 21925 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:27.570659 20646 raft_consensus.cc:1273] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:27.592830 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:27.594626 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:27.595130 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.595999 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.600973 20571 raft_consensus.cc:1273] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:27.601506 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.602960 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:27.675961 20651 raft_consensus.cc:2391] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:27.603217 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.556682 21609 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:27.627760 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:27.551621 21855 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.636765 20575 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3. Ops: []
W20250831 13:58:27.606009 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:27.676831 20643 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:27.692860 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.605458 20616 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:27.626073 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:27.696152 20650 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:27.580161 21751 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:27.700002 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:27.701421 20647 raft_consensus.cc:2366] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:27.706627 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.708037 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.709354 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:27.707544 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 406042us
I20250831 13:58:27.711072 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.710091 20651 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:27.723963 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:27.567337 21783 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: NON_VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: true } }
I20250831 13:58:27.675218 21583 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:27.726222 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:27.677706 21659 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:27.735797 20571 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:27.738662 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:27.727283 20649 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:27.738233 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.737370 20651 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:27.742569 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:27.636811 21858 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.728917 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:27.716760 21896 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.005s sys 0.026s
I20250831 13:58:27.743340 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:27.765728 20645 raft_consensus.cc:1273] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:27.765600 21767 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:27.699584 21508 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.741613 21599 raft_consensus.cc:695] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:27.766765 21896 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s user 0.006s sys 0.026s
I20250831 13:58:27.724911 21751 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:27.791128 20569 consensus_meta.cc:287] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.541s user 0.008s sys 0.002s
W20250831 13:58:27.796684 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:27.799930 20569 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
W20250831 13:58:27.790894 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 185516us
I20250831 13:58:27.751425 21659 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.707504 21925 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:27.800923 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:27.804467 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 50280.3us
I20250831 13:58:27.811370 20616 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:27.814992 20497 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:27.744010 21583 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.791532 21952 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:27.828787 20651 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:27.831363 21956 log.cc:927] Time spent T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20250831 13:58:27.831305 21555 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:27.833068 20467 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:27.833237 21473 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:27.819308 21645 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:27.834043 21645 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:27.832968 21599 consensus_queue.cc:237] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.818964 21977 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:27.796841 21767 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.836580 21645 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:27.832996 21751 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:27.845770 21783 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:27.839018 21931 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:27.846473 21715 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:27.834059 21473 consensus_queue.cc:1046] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
W20250831 13:58:27.938489 21966 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20250831 13:58:27.939543 21975 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 13:58:27.939970 21967 log.cc:927] Time spent T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20250831 13:58:27.940052 21975 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:27.940572 21975 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:27.837421 21952 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:27.910383 21961 log.cc:927] Time spent T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20250831 13:58:27.846329 21896 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20250831 13:58:27.915413 21931 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:27.945606 21919 consensus_meta.cc:287] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.756s user 0.005s sys 0.003s
I20250831 13:58:27.943967 21609 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:27.948734 21959 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.137s user 0.001s sys 0.001s
I20250831 13:58:27.944433 21855 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:27.918973 21556 consensus_queue.cc:1035] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
W20250831 13:58:28.010592 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 161888us
W20250831 13:58:27.954444 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 143163us
I20250831 13:58:27.845912 21977 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:27.955274 21931 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:28.044204 21974 log.cc:927] Time spent T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20250831 13:58:28.045850 21952 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:28.046373 21933 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.915s user 0.000s sys 0.035s
W20250831 13:58:28.048802 21945 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.946s user 0.016s sys 0.012s
I20250831 13:58:28.050287 21977 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:28.051179 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 913) took 1225 ms. Trace:
I20250831 13:58:28.051946 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:28.052440 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 869) took 1145 ms. Trace:
I20250831 13:58:28.052830 20644 rpcz_store.cc:276] 0831 13:58:26.906832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:26.929931 (+ 23099us) service_pool.cc:225] Handling call
0831 13:58:26.948438 (+ 18507us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:26.959961 (+ 11523us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:26.959991 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:26.960485 (+ 494us) log.cc:844] Serialized 26 byte log entry
0831 13:58:27.078360 (+117875us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:27.078392 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:27.078454 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:28.048185 (+969731us) spinlock_profiling.cc:243] Waited 55.7 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:28.052115 (+ 3930us) raft_consensus.cc:1683] finished
0831 13:58:28.052139 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:28.052380 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2242,"raft.queue_time_us":3695,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":128088448,"thread_start_us":99158,"threads_started":1,"wal-append.queue_time_us":142016}
I20250831 13:58:28.051569 20572 rpcz_store.cc:276] 0831 13:58:26.825386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:26.854767 (+ 29381us) service_pool.cc:225] Handling call
0831 13:58:26.856663 (+ 1896us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:26.871257 (+ 14594us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:26.871284 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:26.871764 (+ 480us) log.cc:844] Serialized 26 byte log entry
0831 13:58:26.972075 (+100311us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:27.225872 (+253797us) log.cc:844] Serialized 14 byte log entry
0831 13:58:27.251699 (+ 25827us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:27.251807 (+ 108us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:28.045778 (+793971us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:28.050756 (+ 4978us) raft_consensus.cc:1683] finished
0831 13:58:28.050871 (+ 115us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:28.051006 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21029,"prepare.queue_time_us":219781,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":62493,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":155,"spinlock_wait_cycles":64172416,"thread_start_us":280576,"threads_started":2,"wal-append.queue_time_us":215460}
I20250831 13:58:28.056376 21921 raft_consensus.cc:2953] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Committing config change with OpId 5.5: config changed from index 2 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:28.057825 21941 consensus_queue.cc:1035] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:28.059492 21941 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:28.065034 20616 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:28.070534 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 261214us
I20250831 13:58:27.963354 21919 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.082747 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:28.083359 21914 raft_consensus.cc:3053] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:28.083458 20574 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:28.083770 21914 raft_consensus.cc:738] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:28.084335 21919 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:28.084635 21914 consensus_queue.cc:260] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.031208 21858 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:28.034132 21508 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:28.061056 21925 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:28.107427 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 273227us
W20250831 13:58:28.122231 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 50603.5us
I20250831 13:58:28.128894 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.095129 21980 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.136058 20616 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.116703 21659 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:28.144023 21937 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.991s user 0.005s sys 0.008s
I20250831 13:58:28.117092 21858 consensus_queue.cc:1035] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250831 13:58:28.129470 21583 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:28.195192 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 169592us
I20250831 13:58:28.197201 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 911) took 1377 ms. Trace:
W20250831 13:58:28.211778 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:28.236730 20467 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:28.213184 20577 rpcz_store.cc:276] 0831 13:58:26.819636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:26.831531 (+ 11895us) service_pool.cc:225] Handling call
0831 13:58:26.832180 (+ 649us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:26.850865 (+ 18685us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:26.850905 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:26.872524 (+ 21619us) log.cc:844] Serialized 572 byte log entry
0831 13:58:27.038418 (+165894us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:27.038469 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:27.038574 (+ 105us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:28.139109 (+1100535us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:28.146887 (+ 7778us) raft_consensus.cc:1683] finished
0831 13:58:28.146907 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:28.197120 (+ 50213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47100672,"thread_start_us":165545,"threads_started":1,"wal-append.queue_time_us":238544}
I20250831 13:58:28.145514 21463 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:28.262626 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 472328us
I20250831 13:58:28.264433 20541 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:28.271081 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 56092.7us
W20250831 13:58:28.210762 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:28.147804 21767 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:28.138355 21941 consensus_queue.cc:1046] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20250831 13:58:28.148144 21783 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:28.283743 20541 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.300199 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:28.309145 20571 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:28.305346 20569 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:28.309544 20541 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.318060 20541 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.330482 20572 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:28.214226 21596 consensus_queue.cc:1035] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
W20250831 13:58:28.306365 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:28.140763 21980 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.299547 21582 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:28.299922 21659 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:28.317473 21919 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:28.343640 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:28.344250 20644 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:28.323666 21737 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:28.326962 21609 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:28.328176 21715 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.304729 21556 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20250831 13:58:28.373858 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:28.367971 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1033) took 1136 ms. Trace:
I20250831 13:58:28.339512 21588 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:28.338963 21765 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:28.378852 20574 rpcz_store.cc:276] 0831 13:58:27.210769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:27.281424 (+ 70655us) service_pool.cc:225] Handling call
0831 13:58:28.346718 (+1065294us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b54001a6dd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:28.346881 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1842026752}
I20250831 13:58:28.381840 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:28.385324 20644 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:28.388695 20495 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:58:28.430598 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 93917.5us
I20250831 13:58:28.431598 20541 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.438279 20644 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:28.364692 21659 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.386113 21588 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.468641 20541 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:28.532150 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 63636.3us
I20250831 13:58:28.533403 20541 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.439219 21609 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:28.442405 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:28.390951 21715 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.391342 21765 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.393342 21923 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20250831 13:58:28.467351 21582 raft_consensus.cc:695] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:28.469344 20642 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:28.468291 21977 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:28.472848 21588 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:28.551450 21951 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20250831 13:58:28.554666 21984 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:58:28.566928 21971 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.307s user 0.001s sys 0.001s
I20250831 13:58:28.568878 21528 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:28.569314 21995 raft_consensus.cc:3053] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:28.569828 21995 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:28.571276 21659 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:28.571945 20493 raft_consensus.cc:2408] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:28.601783 21826 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:28.603835 21935 log.cc:927] Time spent T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 1.450s user 0.003s sys 0.010s
I20250831 13:58:28.606140 21210 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47732 (request call id 99) took 1020 ms. Trace:
I20250831 13:58:28.606693 21210 rpcz_store.cc:276] 0831 13:58:27.585329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:27.587316 (+ 1987us) service_pool.cc:225] Handling call
0831 13:58:27.587921 (+ 605us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:27.838121 (+250200us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:27.846625 (+ 8504us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:27.851083 (+ 4458us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:27.853949 (+ 2866us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:28.605839 (+751890us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54005bd1d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:28.605901 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:28.606068 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":539915008}
I20250831 13:58:28.608994 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 910) took 1805 ms. Trace:
I20250831 13:58:28.554651 21991 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:28.609463 20568 rpcz_store.cc:276] 0831 13:58:26.803952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:26.822922 (+ 18970us) service_pool.cc:225] Handling call
0831 13:58:26.823355 (+ 433us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:26.856348 (+ 32993us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:26.856379 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:26.856934 (+ 555us) log.cc:844] Serialized 26 byte log entry
0831 13:58:27.017705 (+160771us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:27.017770 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:27.017853 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:28.602999 (+1585146us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:28.608635 (+ 5636us) raft_consensus.cc:1683] finished
0831 13:58:28.608665 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:28.608923 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27143,"raft.queue_time_us":21458,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":5614,"spinlock_wait_cycles":51110272,"thread_start_us":130607,"threads_started":1,"wal-append.queue_time_us":240290}
I20250831 13:58:28.602358 21826 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.569386 21715 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:28.552824 21609 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.556357 21958 consensus_queue.cc:1046] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.294s
I20250831 13:58:28.618559 21826 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:28.626878 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 301963us
W20250831 13:58:28.631588 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 281423us
I20250831 13:58:28.651780 21997 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
W20250831 13:58:28.654817 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 80834.3us
I20250831 13:58:28.656154 20541 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.662459 20541 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.619401 21991 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:28.672765 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 44314.2us
I20250831 13:58:28.674571 21977 raft_consensus.cc:695] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:28.675024 21919 raft_consensus.cc:695] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:28.675933 21919 consensus_queue.cc:237] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.675931 21977 consensus_queue.cc:237] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.681298 20541 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.682467 20541 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:28.672365 20646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.018s sys 0.035s
I20250831 13:58:28.625514 21582 consensus_queue.cc:237] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.696022 21980 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:28.695096 21930 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.010s sys 0.010s
I20250831 13:58:28.709899 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1021) took 1919 ms. Trace:
I20250831 13:58:28.708631 22013 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250831 13:58:28.694818 21931 consensus_meta.cc:287] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.656s user 0.011s sys 0.000s
I20250831 13:58:28.708858 22020 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:28.714011 20575 rpcz_store.cc:276] 0831 13:58:26.790275 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:26.790804 (+ 529us) service_pool.cc:225] Handling call
0831 13:58:26.827207 (+ 36403us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:27.681274 (+854067us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:28.709618 (+1028344us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b54003b5454. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:28.709832 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1861274496}
W20250831 13:58:28.689296 20646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s user 0.019s sys 0.035s
I20250831 13:58:28.690510 21765 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:28.713577 21930 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s user 0.010s sys 0.010s
I20250831 13:58:28.737659 22023 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:28.738265 22010 log.cc:927] Time spent T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250831 13:58:28.690002 21995 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.713768 21863 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.739650 21995 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:28.739902 21940 log.cc:927] Time spent T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 1.637s user 0.011s sys 0.005s
I20250831 13:58:28.739701 21863 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:28.738356 22023 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.735539 21977 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20250831 13:58:28.779938 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 97541.7us
I20250831 13:58:28.742591 21863 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:28.780934 20541 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:28.774215 21857 consensus_meta.cc:287] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.564s user 0.014s sys 0.000s
I20250831 13:58:28.775002 21977 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.785238 21609 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:28.733371 21931 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:28.873741 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 92896us
I20250831 13:58:28.858129 21919 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:28.877925 20541 leader_election.cc:400] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:28.881003 20541 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.881954 22016 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:28.874349 21977 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:28.883611 21645 consensus_meta.cc:287] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.960s user 0.009s sys 0.003s
I20250831 13:58:28.893096 20541 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.894855 21645 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:28.894891 21958 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:28.897581 21645 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:28.898227 20651 raft_consensus.cc:1238] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:28.878932 21715 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.882930 22024 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:28.904009 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.906893 20541 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.910054 20541 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.911286 21528 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:28.911792 21555 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:28.884310 21991 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:28.912487 22005 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.905262 21659 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:28.913686 21931 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:28.915825 21783 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:28.919508 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 604) took 1315 ms. Trace:
I20250831 13:58:28.917974 21588 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.921949 20541 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:28.905491 21583 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:28.947204 20647 rpcz_store.cc:276] 0831 13:58:27.604311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:27.734217 (+129906us) service_pool.cc:225] Handling call
0831 13:58:27.739050 (+ 4833us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:27.821232 (+ 82182us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:27.821272 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:27.821320 (+ 48us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:27.821370 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:28.919295 (+1097925us) spinlock_profiling.cc:243] Waited 82 ms on lock 0x7b540028a854. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:28.919338 (+ 43us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:28.919454 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":188558592}
I20250831 13:58:28.927771 21826 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.929463 22023 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:28.895066 22022 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:28.955899 22022 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:28.938789 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 557712us
I20250831 13:58:28.908816 21923 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:28.952129 20541 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.954227 22036 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:28.955390 21588 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:28.935560 21555 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:28.959388 22022 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:28.942456 22005 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.950193 21783 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.966647 20541 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.967442 21783 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:28.968847 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.969681 22035 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:28.970034 21555 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:28.970240 22035 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:28.970973 22035 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:28.971436 22035 raft_consensus.cc:695] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:28.972055 21767 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:28.972261 22035 consensus_queue.cc:237] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.972554 21767 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:28.974368 21767 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
W20250831 13:58:28.962142 21921 consensus_meta.cc:287] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.904s user 0.005s sys 0.009s
I20250831 13:58:28.962565 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:28.938306 21645 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:28.958166 21528 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:28.905607 21715 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.975495 20541 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:28.946902 21659 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:28.967443 22005 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:29.019999 20572 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:29.019243 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.019349 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:28.951977 21583 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.008659 21588 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:29.029597 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.026810 21988 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:29.031137 20568 raft_consensus.cc:1273] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:29.031801 20574 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:29.005307 21645 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.032655 21988 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:29.031592 21588 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:29.046380 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.051388 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:29.045742 21988 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:29.043100 21588 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:29.075170 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:29.065579 21975 consensus_meta.cc:287] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 1.125s user 0.014s sys 0.000s
I20250831 13:58:29.081964 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.072968 21984 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:29.085021 21957 log.cc:927] Time spent T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 1.240s user 0.005s sys 0.022s
W20250831 13:58:29.086107 21993 log.cc:927] Time spent T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.604s user 0.000s sys 0.008s
I20250831 13:58:29.099570 20574 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:29.086387 21659 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:29.089330 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.089412 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.091107 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.092859 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.105649 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.107296 20569 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:29.109088 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:29.085453 21977 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.084071 21826 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.114462 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.115628 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.092437 21863 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:29.148032 21941 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:29.153052 21863 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:29.153241 21941 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.148633 21932 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20250831 13:58:29.118439 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.121033 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:29.101455 21960 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:29.125942 21988 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:29.126427 21856 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:29.127096 21715 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:29.128851 21555 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.099957 21984 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.131000 21977 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:29.135303 21998 log.cc:927] Time spent T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.668s user 0.005s sys 0.005s
I20250831 13:58:29.115007 21986 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:29.137751 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:29.138821 22044 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
W20250831 13:58:29.139622 22042 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20250831 13:58:29.112915 21708 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.141332 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:29.118616 22028 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.097s user 0.000s sys 0.001s
W20250831 13:58:29.143512 22030 log.cc:927] Time spent T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.261s user 0.001s sys 0.001s
I20250831 13:58:29.117458 22035 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.143805 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:29.145819 22031 log.cc:927] Time spent T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.261s user 0.001s sys 0.000s
I20250831 13:58:29.145768 21896 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:29.147207 21996 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:29.148401 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 512903us
I20250831 13:58:29.148061 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.151535 21556 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.108544 21659 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.113152 21731 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:29.078989 21975 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.156234 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.270028 21731 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.270552 20400 catalog_manager.cc:5582] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 3 to 4, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to ab7ade51fa134a1b814fab748f01d43d (127.18.233.132). New cstate: current_term: 4 leader_uuid: "ab7ade51fa134a1b814fab748f01d43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:58:29.272126 21731 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.272855 21985 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:29.274938 20575 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:29.268462 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 888) took 1244 ms. Trace:
I20250831 13:58:29.249517 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.145615 21990 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:29.207648 21921 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:29.299829 21583 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:29.375869 21909 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.352085 21659 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.349888 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.352352 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:29.249130 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.371568 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.355816 20642 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250831 13:58:29.205330 21941 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.203516 21863 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:29.201479 21960 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:29.351706 20645 raft_consensus.cc:1238] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:29.275693 20651 rpcz_store.cc:276] 0831 13:58:28.024217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:28.024399 (+ 182us) service_pool.cc:225] Handling call
0831 13:58:28.024707 (+ 308us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:28.898615 (+873908us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:29.268156 (+369541us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b54002ea754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:29.268388 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2008502144}
I20250831 13:58:29.444672 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:29.444826 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.242602 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52966 (request call id 407) took 1156 ms. Trace:
I20250831 13:58:29.445395 20645 raft_consensus.cc:2391] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:29.445566 20494 rpcz_store.cc:276] 0831 13:58:28.085973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:28.086116 (+ 143us) service_pool.cc:225] Handling call
0831 13:58:29.242000 (+1155884us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000113d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8facd0 00007fb9bc97eeea 00007fb9c18d760e 00007fb9baf74cc7 00007fb9baf74c52 00007fb9baf74b81 00007fb9baf74afc 00007fb9baf73db2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:29.242501 (+ 501us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2447096192}
I20250831 13:58:29.207243 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.354614 20650 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:29.387327 20466 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:29.365957 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:29.351159 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.119761 20571 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:58:29.447970 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 60737.9us
I20250831 13:58:29.448273 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.290390 21914 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:29.391058 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:29.450781 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.451582 20645 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:29.451493 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.452723 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.384948 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:29.453449 20645 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:29.453799 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:29.454257 20645 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:29.365451 21932 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:29.399302 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 49056.1us
I20250831 13:58:29.349054 21988 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.348054 21588 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.469825 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.374818 22035 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.357714 21556 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.426719 20575 raft_consensus.cc:2391] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:29.438984 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.421824 20498 raft_consensus.cc:2408] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:29.394670 21562 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.351153 21896 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.422366 21890 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:29.387977 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.449579 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47746 (request call id 48) took 2143 ms. Trace:
I20250831 13:58:29.449831 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.444948 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.498903 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:29.350564 20569 raft_consensus.cc:2391] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:29.456393 21941 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:29.447407 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.396224 21856 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:29.456683 21863 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20250831 13:58:29.483428 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.488847 21562 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.443569 21990 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.503337 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.524994 20497 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:29.443290 21921 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.503015 21941 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:29.503595 20574 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:29.470500 21932 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.527096 21941 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:29.529911 21932 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.489627 21985 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20250831 13:58:29.394969 21596 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:29.440241 20642 pending_rounds.cc:85] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Aborting all ops after (but not including) 2
I20250831 13:58:29.426316 20644 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 4
I20250831 13:58:29.477298 20573 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 1.
I20250831 13:58:29.470862 20570 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 3.
I20250831 13:58:29.503432 20494 raft_consensus.cc:2408] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:29.509397 21960 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:29.503226 20568 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:29.567816 20569 raft_consensus.cc:1238] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:29.569845 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:29.570571 20650 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:29.375880 21708 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.439270 21826 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.480484 20649 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20250831 13:58:29.396740 21976 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.000s sys 0.011s
I20250831 13:58:29.538947 20642 pending_rounds.cc:107] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 13:58:29.376124 21996 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:29.596941 20616 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.130:44629 timed out after 2.230s (SENT) [suppressed 5 similar messages]
I20250831 13:58:29.635754 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 935) took 1304 ms. Trace:
I20250831 13:58:29.473999 21992 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.375612 21986 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:29.392082 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.512831 22007 consensus_queue.cc:1046] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.079s
W20250831 13:58:29.393502 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:29.478753 21565 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20250831 13:58:29.509313 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.510658 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.508783 22035 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.575713 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.527757 21896 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.544155 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.511653 21984 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:29.442528 21909 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.523015 21975 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:29.519223 21562 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.517660 21856 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:29.513334 22008 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.186s
I20250831 13:58:29.643734 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.636133 20577 rpcz_store.cc:276] 0831 13:58:28.331420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:28.340453 (+ 9033us) service_pool.cc:225] Handling call
0831 13:58:28.340903 (+ 450us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:28.674786 (+333883us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:28.674824 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:28.675356 (+ 532us) log.cc:844] Serialized 26 byte log entry
0831 13:58:28.736643 (+ 61287us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:28.736700 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:28.918965 (+182265us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54000b36d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:28.919017 (+ 52us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:29.574691 (+655674us) raft_consensus.cc:1683] finished
0831 13:58:29.574717 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:29.635697 (+ 60980us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9013,"raft.queue_time_us":86013,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":9424,"spinlock_wait_cycles":538894720,"thread_start_us":158619,"threads_started":2,"wal-append.queue_time_us":260341}
I20250831 13:58:29.548512 21596 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.649711 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.522756 21588 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.516268 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:29.520397 21176 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:29.521653 21555 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.510442 21556 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:29.513482 21995 consensus_queue.cc:1046] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.129s
I20250831 13:58:29.504532 21890 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:29.648409 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.650466 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.998950 20651 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:30.005437 20498 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:30.009306 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:30.012691 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:30.014539 20650 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:30.001386 20649 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:30.019407 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:30.020483 20650 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:30.021988 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.029699 21206 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
W20250831 13:58:30.030138 22050 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.373s user 0.001s sys 0.000s
I20250831 13:58:29.541733 21921 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.032610 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:30.048332 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.049722 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:30.032709 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:29.516167 21659 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:29.515268 22021 log.cc:927] Time spent T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.633s user 0.004s sys 0.000s
I20250831 13:58:29.524786 21990 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.014176 21931 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:29.858111 21983 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:29.997155 22048 log.cc:927] Time spent T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.355s user 0.000s sys 0.000s
I20250831 13:58:30.048974 20498 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:29.596586 21583 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.063109 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:29.651093 20642 raft_consensus.cc:2887] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:30.065826 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.695686 21580 consensus_queue.cc:1046] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.435s
I20250831 13:58:29.866454 21731 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:29.978535 21599 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:29.597203 21976 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.715s user 0.000s sys 0.012s
I20250831 13:58:30.014497 21932 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:29.875340 21988 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:29.978205 21907 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:30.092353 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:30.148181 20570 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:29.695308 22003 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:29.600600 21826 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.109964 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:29.652156 21767 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.427s
I20250831 13:58:30.066032 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.157253 20494 raft_consensus.cc:2408] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:30.040546 21984 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.054867 21960 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.124347 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.162254 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.120048 21582 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:29.694775 21992 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:30.140200 20400 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.659s user 0.025s sys 0.013s
I20250831 13:58:30.101318 21896 raft_consensus.cc:3053] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
W20250831 13:58:30.142169 21827 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.030s sys 0.028s
I20250831 13:58:30.127092 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.203760 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:30.130190 21708 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:30.168615 20647 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.020s sys 0.039s
I20250831 13:58:30.169524 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.126773 21985 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:30.215962 20645 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.008s sys 0.011s
I20250831 13:58:30.174511 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:30.216413 20645 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.732s user 0.008s sys 0.011s
I20250831 13:58:30.216624 20570 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:30.100809 21983 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.107400 21931 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:30.203061 21827 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s user 0.030s sys 0.028s
I20250831 13:58:30.114573 21583 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.152959 21907 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.242480 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:30.214025 20644 consensus_meta.cc:287] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.676s user 0.006s sys 0.001s
I20250831 13:58:30.243451 20644 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
W20250831 13:58:30.175455 20400 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.694s user 0.025s sys 0.013s
I20250831 13:58:30.140343 21731 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.148433 21932 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.199864 20576 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:30.142334 21599 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.230966 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.235608 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.199120 21988 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:30.209164 20647 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s user 0.020s sys 0.039s
I20250831 13:58:30.239105 20499 raft_consensus.cc:2391] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:30.239176 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:30.269171 20575 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:30.269845 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.233671 21580 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:30.179657 21896 raft_consensus.cc:738] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:30.168336 21981 consensus_queue.cc:1046] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.060s
I20250831 13:58:30.204774 21996 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.155858 22003 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.251824 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:30.242609 21708 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.175951 21960 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:30.253299 21645 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:30.253652 21975 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:30.250715 20501 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:30.254798 22056 raft_consensus.cc:3053] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Stepping down as leader of term 4
W20250831 13:58:30.255767 20568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1043) took 1174 ms (client timeout 1536 ms). Trace:
I20250831 13:58:30.256309 21932 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.256573 21983 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:30.257773 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:30.259227 20573 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:30.277215 21890 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.264410 21599 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.239184 20541 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:30.351195 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 112082us
I20250831 13:58:30.352308 20541 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:30.353350 20541 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:30.168532 21984 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.271733 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.268453 21988 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.275828 20576 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:30.276141 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.184454 21582 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.358974 21896 consensus_queue.cc:260] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:30.360232 21582 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.324638 21986 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:30.360776 21896 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:30.360980 21986 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:30.326503 22003 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.327170 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:30.362950 20644 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:30.245755 21555 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.366979 21555 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.369336 21555 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.332152 21909 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.277845 20651 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
W20250831 13:58:30.336350 20568 rpcz_store.cc:269] 0831 13:58:29.081274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.081662 (+ 388us) service_pool.cc:225] Handling call
0831 13:58:30.255592 (+1173930us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54002ff154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:30.255713 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":963831680}
I20250831 13:58:29.497571 21205 rpcz_store.cc:276] 0831 13:58:27.314139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:27.318837 (+ 4698us) service_pool.cc:225] Handling call
0831 13:58:27.497386 (+178549us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:27.581800 (+ 84414us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:27.589425 (+ 7625us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:27.740220 (+150795us) log.cc:844] Serialized 26 byte log entry
0831 13:58:28.316543 (+576323us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:28.316635 (+ 92us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:28.316753 (+ 118us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:29.134673 (+817920us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:29.398900 (+264227us) raft_consensus.cc:1683] finished
0831 13:58:29.398924 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:29.449469 (+ 50545us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6203,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":10286,"spinlock_wait_cycles":3612544,"thread_start_us":484219,"threads_started":1,"wal-append.queue_time_us":602954}
I20250831 13:58:30.271839 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:30.341043 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:30.262195 21941 consensus_meta.cc:287] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.731s user 0.006s sys 0.002s
I20250831 13:58:30.228366 21985 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.265002 21856 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:30.355859 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.408478 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:30.215615 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 494529us
I20250831 13:58:30.354373 20541 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:30.410106 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 55860.1us
I20250831 13:58:30.328508 21596 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.359762 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.248160 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 881) took 2522 ms. Trace:
I20250831 13:58:30.335736 21565 raft_consensus.cc:3053] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:30.285126 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:30.335985 21645 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:30.246090 21609 consensus_queue.cc:1046] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.093s
I20250831 13:58:30.370941 21977 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:30.336344 22056 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:30.322389 21580 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:30.380391 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.738s user 0.025s sys 0.051s
W20250831 13:58:30.397915 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:30.435322 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.793s user 0.026s sys 0.055s
W20250831 13:58:30.435779 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 793152us
I20250831 13:58:30.378062 20644 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:30.406240 20573 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:30.436805 21983 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:30.437312 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.411329 20541 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:30.437876 20616 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:30.438898 20541 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:30.438306 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.440322 20570 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:30.441314 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.442010 20497 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:30.409180 20494 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:30.334189 21708 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.437898 20573 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:30.357213 21588 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:30.451150 21588 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:30.371982 22035 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:30.422413 21960 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:30.424860 21556 raft_consensus.cc:3053] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:30.426393 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 940) took 1049 ms. Trace:
I20250831 13:58:30.427652 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1049) took 1036 ms. Trace:
I20250831 13:58:30.428001 21609 raft_consensus.cc:3053] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:30.337195 21975 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.443567 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.444811 20616 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:30.417146 21555 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:30.445166 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.447867 21983 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:30.450229 21708 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.435189 21645 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:30.418440 20642 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:30.416329 21565 raft_consensus.cc:738] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:30.336149 20571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.015s sys 0.041s
W20250831 13:58:30.478953 20571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.015s sys 0.041s
I20250831 13:58:30.460363 21609 raft_consensus.cc:738] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:30.454068 20467 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:30.480319 20467 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:30.434497 21977 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:30.455448 20576 rpcz_store.cc:276] 0831 13:58:29.377205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.384684 (+ 7479us) service_pool.cc:225] Handling call
0831 13:58:30.426167 (+1041483us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b54001a5ed4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:30.426311 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1675506560}
I20250831 13:58:30.484532 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.486419 21909 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:30.489969 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 939) took 1122 ms. Trace:
I20250831 13:58:30.455572 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.455530 20496 rpcz_store.cc:276] 0831 13:58:29.393011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.393137 (+ 126us) service_pool.cc:225] Handling call
0831 13:58:29.424068 (+ 30931us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:29.527262 (+103194us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:29.527291 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:29.527341 (+ 50us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:29.527369 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:30.427350 (+899981us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54001c0f54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:30.427389 (+ 39us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:30.427500 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236919168}
I20250831 13:58:30.435678 21826 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.490469 20569 rpcz_store.cc:276] 0831 13:58:29.367117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.568801 (+201684us) service_pool.cc:225] Handling call
0831 13:58:29.610344 (+ 41543us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:29.652960 (+ 42616us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:29.652989 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:29.734746 (+ 81757us) log.cc:844] Serialized 26 byte log entry
0831 13:58:30.120773 (+386027us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:30.120815 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:30.120885 (+ 70us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:30.489720 (+368835us) raft_consensus.cc:1683] finished
0831 13:58:30.489747 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:30.489902 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7078,"raft.queue_time_us":7245,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":61510,"thread_start_us":364426,"threads_started":1,"wal-append.queue_time_us":719039}
I20250831 13:58:30.465893 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.495059 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.419816 21856 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:30.451785 21596 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:30.491122 21977 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:30.430881 21941 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.473637 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:30.501965 20543 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s user 0.033s sys 0.054s
I20250831 13:58:30.453455 22035 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.458743 21556 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:30.507272 20543 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s user 0.034s sys 0.054s
W20250831 13:58:30.464643 21930 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.034s sys 0.065s
I20250831 13:58:30.463943 21960 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:30.464969 21992 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.493005 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:30.491518 20497 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:30.472654 21931 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.472062 20573 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:30.474583 21983 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.477665 21907 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.515051 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.502183 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.515960 20496 raft_consensus.cc:2366] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4.
I20250831 13:58:30.504494 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.515153 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.539412 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.501230 21583 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:30.504947 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.508781 22035 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:30.496573 21731 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.511087 20498 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:30.494256 21932 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:30.516686 20570 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:30.496924 21909 consensus_queue.cc:1046] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
W20250831 13:58:30.585831 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 92905.4us
I20250831 13:58:30.557727 20543 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:30.553134 21983 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:30.479709 22056 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:30.569638 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:30.553521 21930 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.037s user 0.035s sys 0.065s
I20250831 13:58:30.581004 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.501266 21596 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.470077 20568 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:30.602466 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:30.553874 21555 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.498880 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.629813 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:30.632838 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 75223.3us
I20250831 13:58:30.638037 20497 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:30.670668 20543 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:30.672748 20543 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:30.591215 21932 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:30.676601 21932 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:30.568735 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:30.499516 21565 consensus_queue.cc:260] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.678325 20569 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:30.679680 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 49986.2us
I20250831 13:58:30.680963 20467 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:30.746428 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:30.640460 21979 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.012s sys 0.037s
W20250831 13:58:30.883901 21979 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.832s user 0.013s sys 0.040s
I20250831 13:58:30.883812 21596 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.885027 21979 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:30.885246 20569 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:30.679713 21175 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:30.886072 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 944) took 1447 ms. Trace:
W20250831 13:58:30.886184 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 245783us
I20250831 13:58:30.639565 21931 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:30.680645 21555 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.508486 21609 consensus_queue.cc:260] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.631237 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:30.640254 21984 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:30.891436 21205 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 1.
I20250831 13:58:30.892571 21984 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:30.763818 20570 raft_consensus.cc:2953] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 2 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
W20250831 13:58:30.630467 21176 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s user 0.014s sys 0.019s
I20250831 13:58:30.882876 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:30.639596 21960 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:30.586009 21985 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:30.688910 21580 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:30.746510 21992 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:30.630892 21731 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:30.581835 21583 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.103s
I20250831 13:58:30.671911 21556 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:30.886519 20569 rpcz_store.cc:276] 0831 13:58:29.439144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.746833 (+1307689us) service_pool.cc:225] Handling call
0831 13:58:30.885026 (+138193us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:30.885780 (+ 754us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:30.886007 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:30.909279 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:30.746309 21896 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:30.887375 21931 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:30.773087 22056 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:30.677225 21978 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.140s user 0.030s sys 0.102s
W20250831 13:58:30.886651 21914 consensus_meta.cc:287] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 1.419s user 0.005s sys 0.005s
W20250831 13:58:30.931250 20645 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.018s sys 0.043s
W20250831 13:58:31.079684 20645 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.863s user 0.018s sys 0.043s
I20250831 13:58:30.416999 20646 rpcz_store.cc:276] 0831 13:58:27.712591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:27.755941 (+ 43350us) service_pool.cc:225] Handling call
0831 13:58:27.813314 (+ 57373us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:27.851969 (+ 38655us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:27.853677 (+ 1708us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:27.956812 (+103135us) log.cc:844] Serialized 26 byte log entry
0831 13:58:28.599126 (+642314us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9b99f739c 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb
0831 13:58:28.768765 (+169639us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:28.770638 (+ 1873us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:28.781191 (+ 10553us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:29.389418 (+608227us) spinlock_profiling.cc:243] Waited 1.76 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:30.208106 (+818688us) raft_consensus.cc:1683] finished
0831 13:58:30.215861 (+ 7755us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:30.235115 (+ 19254us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1105,"raft.queue_time_us":102746,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":69,"spinlock_wait_cycles":47913600,"thread_start_us":686290,"threads_started":1,"wal-append.queue_time_us":845876}
I20250831 13:58:30.940659 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 937) took 1662 ms. Trace:
I20250831 13:58:30.893087 21984 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:30.888875 21645 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:30.888645 21975 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:30.915663 21210 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:30.669533 20494 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:30.927738 21992 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:31.073989 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 942) took 1682 ms. Trace:
I20250831 13:58:31.113905 20568 rpcz_store.cc:276] 0831 13:58:29.391241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.435549 (+1044308us) service_pool.cc:225] Handling call
0831 13:58:30.435964 (+ 415us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.065891 (+629927us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.073944 (+ 8053us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.079617 21609 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:30.886821 21588 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
W20250831 13:58:30.914716 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 1.3943e+06us
I20250831 13:58:31.080405 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 904) took 1726 ms. Trace:
I20250831 13:58:31.098157 21984 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:31.129412 20645 rpcz_store.cc:276] 0831 13:58:29.353571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.466417 (+112846us) service_pool.cc:225] Handling call
0831 13:58:29.466793 (+ 376us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:29.467020 (+ 227us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:30.216910 (+749890us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.080057 (+863147us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.080110 (+ 53us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.080162 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.080234 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.080354 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":398757,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":18877,"raft.queue_time_us":27179,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":3869,"thread_start_us":1595208,"threads_started":2}
W20250831 13:58:31.135510 20614 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.193s user 0.014s sys 0.061s
W20250831 13:58:31.136013 20614 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.262s user 0.015s sys 0.061s
W20250831 13:58:30.928666 21827 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.013s sys 0.046s
W20250831 13:58:31.139283 21827 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.886s user 0.013s sys 0.054s
I20250831 13:58:30.887133 21175 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:58:31.140043 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 253030us
I20250831 13:58:31.084070 20577 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:31.086469 20571 rpcz_store.cc:276] 0831 13:58:29.278009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.478881 (+200872us) service_pool.cc:225] Handling call
0831 13:58:29.485540 (+ 6659us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:29.504228 (+ 18688us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:29.504273 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:29.569374 (+ 65101us) log.cc:844] Serialized 26 byte log entry
0831 13:58:30.479362 (+909988us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:30.479401 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:30.479464 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:30.940420 (+460956us) raft_consensus.cc:1683] finished
0831 13:58:30.940448 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:30.940592 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11025,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":721091,"threads_started":1,"wal-append.queue_time_us":958832}
I20250831 13:58:31.142644 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 949) took 1095 ms. Trace:
I20250831 13:58:31.143069 20571 rpcz_store.cc:276] 0831 13:58:30.047163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.141659 (+1094496us) service_pool.cc:225] Handling call
0831 13:58:31.142050 (+ 391us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.142251 (+ 201us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.142298 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.142350 (+ 52us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:31.142391 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.142462 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.142583 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.145025 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 950) took 1090 ms. Trace:
I20250831 13:58:31.145413 20571 rpcz_store.cc:276] 0831 13:58:30.054202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.144144 (+1089942us) service_pool.cc:225] Handling call
0831 13:58:31.144482 (+ 338us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.144663 (+ 181us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.144696 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.144747 (+ 51us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.144786 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.144850 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.144967 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:30.990846 22059 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.485s user 0.000s sys 0.001s
I20250831 13:58:31.101539 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:31.147460 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 619) took 1774 ms. Trace:
W20250831 13:58:31.081459 21978 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.548s user 0.038s sys 0.111s
I20250831 13:58:30.681988 20467 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:31.148721 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 466773us
I20250831 13:58:31.147790 20647 rpcz_store.cc:276] 0831 13:58:29.373357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.502040 (+128683us) service_pool.cc:225] Handling call
0831 13:58:29.503548 (+ 1508us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:29.510105 (+ 6557us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:29.510138 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:29.510662 (+ 524us) log.cc:844] Serialized 26 byte log entry
0831 13:58:30.268535 (+757873us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:30.268581 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:30.268652 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:31.147267 (+878615us) raft_consensus.cc:1683] finished
0831 13:58:31.147292 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.147407 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4823,"raft.queue_time_us":783,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":757510,"threads_started":1,"wal-append.queue_time_us":965664}
I20250831 13:58:30.936800 21986 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:31.151000 20647 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20250831 13:58:30.928928 21976 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.009s sys 0.050s
I20250831 13:58:31.082816 21896 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:31.151933 22003 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
W20250831 13:58:31.152019 21976 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.989s user 0.009s sys 0.050s
I20250831 13:58:31.152129 21896 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:31.152650 22003 raft_consensus.cc:695] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:31.020080 21960 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:31.078841 21931 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:31.153522 22003 consensus_queue.cc:237] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.097430 21565 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:30.937180 20497 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:31.093658 21856 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:31.092759 21985 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:31.098340 21914 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:31.163908 21960 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:31.164177 21173 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:31.132419 21556 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:31.164440 21960 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.166311 21960 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.166098 21896 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.118070 21941 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.170207 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 951) took 1066 ms. Trace:
I20250831 13:58:30.923708 21979 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.150997 21986 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:31.118973 20575 consensus_meta.cc:287] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.843s user 0.010s sys 0.000s
I20250831 13:58:31.175043 20575 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:31.175654 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1073) took 1578 ms. Trace:
I20250831 13:58:31.176020 20575 rpcz_store.cc:276] 0831 13:58:29.597494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.215561 (+618067us) service_pool.cc:225] Handling call
0831 13:58:31.175593 (+960032us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:31.118561 21747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.792s user 0.012s sys 0.061s
I20250831 13:58:31.107352 21930 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:31.118366 21992 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:31.161098 20467 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:31.162626 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 631) took 1522 ms. Trace:
I20250831 13:58:31.192494 20647 rpcz_store.cc:276] 0831 13:58:29.640707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.150282 (+1509575us) service_pool.cc:225] Handling call
0831 13:58:31.150608 (+ 326us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.161843 (+ 11235us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.162556 (+ 713us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:31.193641 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:31.188719 21747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.865s user 0.013s sys 0.061s
W20250831 13:58:31.172075 20541 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.029s sys 0.040s
I20250831 13:58:31.124943 21555 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.169962 22067 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:31.201754 20647 raft_consensus.cc:1273] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:31.202494 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 633) took 1345 ms. Trace:
I20250831 13:58:31.202873 20647 rpcz_store.cc:276] 0831 13:58:29.857011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.197886 (+1340875us) service_pool.cc:225] Handling call
0831 13:58:31.201309 (+ 3423us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.202188 (+ 879us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.202429 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:31.204396 20644 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.019s sys 0.025s
W20250831 13:58:31.205344 20644 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s user 0.019s sys 0.025s
I20250831 13:58:31.207778 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 909) took 1681 ms. Trace:
I20250831 13:58:31.209980 20644 rpcz_store.cc:276] 0831 13:58:29.525989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.441544 (+915555us) service_pool.cc:225] Handling call
0831 13:58:30.441853 (+ 309us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:30.442070 (+ 217us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:30.474921 (+ 32851us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.207366 (+732445us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.207435 (+ 69us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.207490 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.207581 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.207717 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":32662,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"thread_start_us":731437,"threads_started":1}
I20250831 13:58:31.172593 22069 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:31.223830 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 639) took 1089 ms. Trace:
I20250831 13:58:31.231062 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 636) took 1200 ms. Trace:
I20250831 13:58:31.195768 21941 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:31.229632 20644 rpcz_store.cc:276] 0831 13:58:30.134722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.222909 (+1088187us) service_pool.cc:225] Handling call
0831 13:58:31.223279 (+ 370us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.223465 (+ 186us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.223494 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.223549 (+ 55us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.223577 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.223642 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.223774 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.188529 21985 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.240969 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 640) took 1067 ms. Trace:
I20250831 13:58:31.242892 20644 rpcz_store.cc:276] 0831 13:58:30.174080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.237864 (+1063784us) service_pool.cc:225] Handling call
0831 13:58:31.240354 (+ 2490us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.240589 (+ 235us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.240627 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.240685 (+ 58us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.240726 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.240783 (+ 57us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.240907 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.148016 20496 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250831 13:58:31.188894 20643 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.307s user 0.024s sys 0.051s
W20250831 13:58:31.244253 20643 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.379s user 0.024s sys 0.052s
I20250831 13:58:31.231492 20647 rpcz_store.cc:276] 0831 13:58:30.030227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.221009 (+1190782us) service_pool.cc:225] Handling call
0831 13:58:31.230389 (+ 9380us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.230741 (+ 352us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.230789 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.230829 (+ 40us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.230856 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.230905 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.231014 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:30.993763 21731 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.246810 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 620) took 1873 ms. Trace:
I20250831 13:58:31.247270 20643 rpcz_store.cc:276] 0831 13:58:29.373658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.708381 (+334723us) service_pool.cc:225] Handling call
0831 13:58:29.760395 (+ 52014us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:29.760668 (+ 273us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:29.864810 (+104142us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.244772 (+1379962us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.244829 (+ 57us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.244879 (+ 50us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.244954 (+ 75us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.246746 (+ 1792us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20987,"prepare.queue_time_us":254000,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":6723,"raft.queue_time_us":574,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":1379566,"threads_started":1}
I20250831 13:58:31.168144 21896 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:31.196884 20541 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s user 0.030s sys 0.041s
W20250831 13:58:31.249373 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 809010us
W20250831 13:58:31.249764 20642 consensus_meta.cc:287] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.751s user 0.008s sys 0.004s
I20250831 13:58:31.250103 20642 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
W20250831 13:58:31.243672 20498 consensus_meta.cc:287] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.668s user 0.008s sys 0.000s
I20250831 13:58:31.250135 20541 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
W20250831 13:58:31.250771 20650 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.657s user 0.000s sys 0.018s
I20250831 13:58:31.250944 20498 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:31.171165 20571 rpcz_store.cc:276] 0831 13:58:30.103861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.160865 (+1057004us) service_pool.cc:225] Handling call
0831 13:58:31.168925 (+ 8060us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.169309 (+ 384us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.169462 (+ 153us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.169583 (+ 121us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:31.169662 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.169921 (+ 259us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.170117 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.251441 21555 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:31.192858 21930 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.252789 20467 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:31.263259 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 70925us
I20250831 13:58:31.251935 20541 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:31.264816 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:31.251883 20650 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.000s sys 0.019s
I20250831 13:58:31.266520 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 624) took 1891 ms. Trace:
I20250831 13:58:31.266782 20541 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:31.254401 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1060) took 1744 ms. Trace:
I20250831 13:58:31.237599 21941 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:31.234632 22070 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:31.570149 20496 rpcz_store.cc:276] 0831 13:58:29.509503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.147303 (+1637800us) service_pool.cc:225] Handling call
0831 13:58:31.147642 (+ 339us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.244162 (+ 96520us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.254334 (+ 10172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.346393 21583 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:31.347131 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 82378.4us
I20250831 13:58:31.228392 22067 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:31.250682 20642 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 637) took 1220 ms (client timeout 1624 ms). Trace:
I20250831 13:58:31.543105 20571 raft_consensus.cc:1273] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:31.579005 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 968) took 1130 ms. Trace:
I20250831 13:58:31.579783 20571 rpcz_store.cc:276] 0831 13:58:30.448309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.424142 (+975833us) service_pool.cc:225] Handling call
0831 13:58:31.542693 (+118551us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.578652 (+ 35959us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.578942 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.365772 20650 rpcz_store.cc:276] 0831 13:58:29.374781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.042413 (+667632us) service_pool.cc:225] Handling call
0831 13:58:30.078571 (+ 36158us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:30.078799 (+ 228us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:30.112003 (+ 33204us) log.cc:844] Serialized 14 byte log entry
0831 13:58:30.593644 (+481641us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.266227 (+672583us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.266286 (+ 59us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:31.266322 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.266378 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.266475 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16286,"prepare.queue_time_us":49542,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":105,"raft.queue_time_us":812980,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":78366,"spinlock_wait_cycles":134713088,"thread_start_us":1153347,"threads_started":2,"wal-append.queue_time_us":1046508}
W20250831 13:58:31.404651 20574 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1065) took 2203 ms (client timeout 2031 ms). Trace:
W20250831 13:58:31.582100 20574 rpcz_store.cc:269] 0831 13:58:29.142621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.155922 (+ 13301us) service_pool.cc:225] Handling call
0831 13:58:31.345336 (+2189414us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54002fe254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:31.345470 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":536163584}
I20250831 13:58:31.274348 21960 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
W20250831 13:58:31.380208 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.905s user 0.007s sys 0.019s
W20250831 13:58:31.587461 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.112s user 0.008s sys 0.020s
W20250831 13:58:31.587838 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 1.143e+06us
I20250831 13:58:31.589843 20616 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:31.269212 21896 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:31.559151 22068 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.365s user 0.000s sys 0.001s
W20250831 13:58:31.396136 20570 consensus_meta.cc:287] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.502s user 0.011s sys 0.002s
I20250831 13:58:31.253638 22069 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:31.271387 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:31.505486 21731 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.574973 21941 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:31.591161 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 324454us
I20250831 13:58:31.610396 20541 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:31.612620 20541 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:31.592432 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 912) took 2065 ms. Trace:
I20250831 13:58:31.603654 20574 raft_consensus.cc:1238] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:31.614189 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 973) took 1164 ms. Trace:
I20250831 13:58:31.574759 22070 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:31.598639 21896 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:31.607482 22069 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:31.615376 20574 rpcz_store.cc:276] 0831 13:58:30.449919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.583283 (+1133364us) service_pool.cc:225] Handling call
0831 13:58:31.603422 (+ 20139us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.613968 (+ 10546us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.614141 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.598888 21960 raft_consensus.cc:695] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:58:31.667025 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 57587.8us
I20250831 13:58:31.593729 21596 consensus_queue.cc:1046] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.070s
W20250831 13:58:31.657940 21173 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:31.613821 20645 rpcz_store.cc:276] 0831 13:58:29.526838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.134054 (+1607216us) service_pool.cc:225] Handling call
0831 13:58:31.134430 (+ 376us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.134673 (+ 243us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:31.170381 (+ 35708us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.591973 (+421592us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.592056 (+ 83us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:31.592111 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.592194 (+ 83us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.592368 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":37071,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":26136,"thread_start_us":404662,"threads_started":1}
W20250831 13:58:31.690110 20568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.017s sys 0.034s
W20250831 13:58:31.690649 20568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s user 0.017s sys 0.035s
I20250831 13:58:31.585112 21583 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:31.587507 20642 rpcz_store.cc:269] 0831 13:58:30.030518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.237175 (+206657us) service_pool.cc:225] Handling call
0831 13:58:31.250630 (+1013455us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.608481 22081 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:31.658303 21979 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:31.712272 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:31.662248 21731 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
W20250831 13:58:31.728833 22073 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.335s user 0.000s sys 0.000s
I20250831 13:58:31.729210 21731 raft_consensus.cc:695] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:31.731003 21985 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.614923 21914 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:31.731611 21985 raft_consensus.cc:3053] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:31.731977 21985 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:31.732721 21985 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.733878 21985 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:31.735220 22062 log.cc:927] Time spent T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.213s user 0.000s sys 0.000s
W20250831 13:58:31.737083 22079 log.cc:927] Time spent T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
W20250831 13:58:31.737844 22072 log.cc:927] Time spent T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.466s user 0.000s sys 0.001s
W20250831 13:58:31.740684 21556 consensus_meta.cc:287] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.576s user 0.007s sys 0.000s
I20250831 13:58:31.735174 21960 consensus_queue.cc:237] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:31.728039 22077 log.cc:927] Time spent T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20250831 13:58:31.730446 21731 consensus_queue.cc:237] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.665458 21896 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:31.740466 21914 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:31.728183 22061 log.cc:927] Time spent T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.463s user 0.000s sys 0.000s
I20250831 13:58:31.658051 22070 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:31.748706 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 170381us
W20250831 13:58:31.659703 21176 connection.cc:232] RPC call timeout handler was delayed by -1.3522364230000221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20250831 13:58:31.747428 21747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.004s sys 0.014s
W20250831 13:58:31.670610 22068 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20250831 13:58:31.759220 20543 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.961s user 0.039s sys 0.064s
W20250831 13:58:31.769387 20543 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.971s user 0.040s sys 0.064s
I20250831 13:58:31.721045 22083 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.084s
W20250831 13:58:31.759078 21747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s user 0.005s sys 0.014s
I20250831 13:58:31.744813 22078 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 13:58:31.759573 22003 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.019s sys 0.026s
I20250831 13:58:31.780017 22069 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.782485 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:31.787830 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 174333us
I20250831 13:58:31.772527 21979 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:31.785290 22003 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s user 0.019s sys 0.027s
I20250831 13:58:31.794754 21979 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.779384 21930 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.751448 21914 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:31.823228 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1059) took 2314 ms. Trace:
I20250831 13:58:31.784945 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 910) took 2258 ms. Trace:
I20250831 13:58:31.826658 20494 rpcz_store.cc:276] 0831 13:58:29.509247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.116981 (+1607734us) service_pool.cc:225] Handling call
0831 13:58:31.117304 (+ 323us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.117515 (+ 211us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:31.598259 (+480744us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.822868 (+224609us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.822922 (+ 54us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:31.822960 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.823028 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.823159 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":145976,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":69664,"thread_start_us":704659,"threads_started":2}
I20250831 13:58:31.787464 20642 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 3.
I20250831 13:58:31.835091 20494 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29"
dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:35300
I20250831 13:58:31.830130 21909 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:31.841060 20494 raft_consensus.cc:604] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:31.843367 20494 raft_consensus.cc:612] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:58:31.844602 21174 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:31.847270 22069 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:31.847648 22069 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:31.848040 22069 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:31.848124 22051 log.cc:927] Time spent T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 1.719s user 0.003s sys 0.005s
I20250831 13:58:31.849735 22100 consensus_queue.cc:1035] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:31.851236 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1050) took 2762 ms. Trace:
I20250831 13:58:31.851948 20572 rpcz_store.cc:276] 0831 13:58:29.089312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.116198 (+ 26886us) service_pool.cc:225] Handling call
0831 13:58:29.116530 (+ 332us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:29.116892 (+ 362us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:29.116925 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:29.123311 (+ 6386us) log.cc:844] Serialized 26 byte log entry
0831 13:58:29.468944 (+345633us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:29.468991 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:29.469055 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:31.847626 (+2378571us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:31.850827 (+ 3201us) raft_consensus.cc:1683] finished
0831 13:58:31.850865 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.851149 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":85,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":2921,"spinlock_wait_cycles":18160384,"thread_start_us":345254,"threads_started":1,"wal-append.queue_time_us":907433}
W20250831 13:58:31.854219 22087 log.cc:927] Time spent T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250831 13:58:31.856130 21979 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.857254 21941 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.831377 20646 rpcz_store.cc:276] 0831 13:58:29.526279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.124561 (+1598282us) service_pool.cc:225] Handling call
0831 13:58:31.193077 (+ 68516us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.216964 (+ 23887us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:31.265566 (+ 48602us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.749937 (+484371us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.760983 (+ 11046us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:31.769820 (+ 8837us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.774265 (+ 4445us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.783554 (+ 9289us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11604,"prepare.run_wall_time_us":3621,"raft.queue_time_us":2428,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":17834,"spinlock_wait_cycles":16486656,"thread_start_us":399827,"threads_started":1,"wal-append.queue_time_us":527327}
I20250831 13:58:31.859261 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 943) took 2434 ms. Trace:
I20250831 13:58:31.860647 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 947) took 1837 ms. Trace:
W20250831 13:58:31.879434 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 209367us
I20250831 13:58:31.879222 20576 rpcz_store.cc:276] 0831 13:58:30.023276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.113094 (+1089818us) service_pool.cc:225] Handling call
0831 13:58:31.113459 (+ 365us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.113729 (+ 270us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:31.604665 (+490936us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.860262 (+255597us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.860338 (+ 76us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:31.860390 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.860470 (+ 80us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.860590 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":637578,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"thread_start_us":699303,"threads_started":2,"wal-append.queue_time_us":239351}
I20250831 13:58:31.880183 20616 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:31.863041 21931 consensus_meta.cc:287] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.709s user 0.007s sys 0.003s
W20250831 13:58:31.865041 21977 consensus_meta.cc:287] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 1.360s user 0.011s sys 0.004s
I20250831 13:58:31.865617 21583 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:31.868824 21907 raft_consensus.cc:3053] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
W20250831 13:58:31.869695 21984 consensus_meta.cc:287] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.699s user 0.001s sys 0.006s
I20250831 13:58:31.822391 22066 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.018s
I20250831 13:58:31.873450 22067 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:31.881433 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 128396us
I20250831 13:58:31.881652 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 617) took 1962 ms. Trace:
I20250831 13:58:31.878232 20570 rpcz_store.cc:276] 0831 13:58:29.424607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.633252 (+1208645us) service_pool.cc:225] Handling call
0831 13:58:30.763438 (+130186us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:30.763682 (+ 244us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 13:58:31.599983 (+836301us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.858875 (+258892us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.858942 (+ 67us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:31.858997 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.859078 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.859206 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14956,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":2262,"thread_start_us":258515,"threads_started":1,"wal-append.queue_time_us":270816}
I20250831 13:58:31.893884 20494 rpcz_store.cc:276] 0831 13:58:29.919503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.880850 (+1961347us) service_pool.cc:225] Handling call
0831 13:58:31.881124 (+ 274us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.881330 (+ 206us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.881356 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.881395 (+ 39us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:31.881422 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.881494 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.881603 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:31.866664 22035 consensus_meta.cc:287] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 1.264s user 0.002s sys 0.009s
I20250831 13:58:31.895364 22035 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.897461 22035 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:31.897876 21565 consensus_meta.cc:287] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.743s user 0.009s sys 0.016s
I20250831 13:58:31.898499 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 948) took 1865 ms. Trace:
I20250831 13:58:31.889174 21977 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.898918 20568 rpcz_store.cc:276] 0831 13:58:30.032698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.114883 (+1082185us) service_pool.cc:225] Handling call
0831 13:58:31.115181 (+ 298us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:31.120252 (+ 5071us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.120276 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:31.124393 (+ 4117us) log.cc:844] Serialized 52 byte log entry
0831 13:58:31.723381 (+598988us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:31.723443 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.723537 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:31.898310 (+174773us) raft_consensus.cc:1683] finished
0831 13:58:31.898327 (+ 17us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.898447 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":48240,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":503375,"threads_started":1,"wal-append.queue_time_us":672725}
I20250831 13:58:31.900055 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 945) took 2458 ms. Trace:
I20250831 13:58:31.900252 21977 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.901062 21659 raft_consensus.cc:3053] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:31.887986 21931 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.900403 20569 rpcz_store.cc:276] 0831 13:58:29.441225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.038489 (+1597264us) service_pool.cc:225] Handling call
0831 13:58:31.345523 (+307034us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:31.392665 (+ 47142us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.393293 (+ 628us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:31.558024 (+164731us) log.cc:844] Serialized 26 byte log entry
0831 13:58:31.743219 (+185195us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.743260 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.743398 (+ 138us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:31.898093 (+154695us) raft_consensus.cc:1683] finished
0831 13:58:31.898112 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.899835 (+ 1723us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54002fe250. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:31.900006 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":76341,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":99360,"spinlock_wait_cycles":575259904,"thread_start_us":184737,"threads_started":1,"wal-append.queue_time_us":278332}
I20250831 13:58:31.891589 20496 raft_consensus.cc:1238] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:31.885943 22081 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:31.887804 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 116346us
I20250831 13:58:31.901575 21659 raft_consensus.cc:738] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:31.889413 21907 raft_consensus.cc:738] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:31.902184 20573 raft_consensus.cc:2391] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:31.903617 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1065) took 2258 ms. Trace:
I20250831 13:58:31.903321 21907 consensus_queue.cc:260] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.903946 20496 rpcz_store.cc:276] 0831 13:58:29.645759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.584263 (+1938504us) service_pool.cc:225] Handling call
0831 13:58:31.604742 (+ 20479us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.902134 (+297392us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.903416 (+ 1282us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b54002df354. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:31.903565 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":659311360}
I20250831 13:58:31.904537 21907 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:31.905575 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 621) took 1984 ms. Trace:
I20250831 13:58:31.905861 20496 rpcz_store.cc:276] 0831 13:58:29.920722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.904805 (+1984083us) service_pool.cc:225] Handling call
0831 13:58:31.905107 (+ 302us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.905271 (+ 164us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.905296 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.905334 (+ 38us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.905365 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.905415 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.905514 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:31.906881 20571 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:31.907100 21659 consensus_queue.cc:260] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.908057 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 970) took 1459 ms. Trace:
I20250831 13:58:31.908243 21659 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:31.908398 20571 rpcz_store.cc:276] 0831 13:58:30.448925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.581076 (+1132151us) service_pool.cc:225] Handling call
0831 13:58:31.599473 (+ 18397us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.907310 (+307837us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.907838 (+ 528us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54000b36d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:31.907992 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":706519808}
W20250831 13:58:31.909088 22056 consensus_meta.cc:287] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.821s user 0.006s sys 0.006s
W20250831 13:58:31.897938 22092 log.cc:927] Time spent T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20250831 13:58:31.874279 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 99826.8us
I20250831 13:58:31.914716 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:31.914954 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:31.915710 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:58:31.873225 21932 consensus_meta.cc:287] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.174s user 0.007s sys 0.004s
I20250831 13:58:31.929240 20572 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:31.902316 20616 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:31.925920 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:31.918049 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:31.914032 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:31.901605 21992 consensus_meta.cc:287] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.706s user 0.011s sys 0.001s
W20250831 13:58:31.882776 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 534068us
I20250831 13:58:31.903407 21984 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:31.891912 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 68747.4us
W20250831 13:58:31.897471 22090 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250831 13:58:31.906900 22096 consensus_queue.cc:1035] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250831 13:58:31.937599 20569 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:31.902254 22081 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:31.923768 21609 consensus_meta.cc:287] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.798s user 0.002s sys 0.008s
I20250831 13:58:31.940486 20574 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:31.945158 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:31.945928 20570 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
W20250831 13:58:31.927449 21983 consensus_meta.cc:287] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 1.236s user 0.010s sys 0.011s
I20250831 13:58:31.947844 20541 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:31.925578 22075 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:31.966254 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:31.967129 20645 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:31.968264 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:31.969502 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:31.971140 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:31.934242 21588 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:31.937546 21767 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:31.972663 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:31.975106 20647 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:31.977700 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:31.978462 20650 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:31.980306 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:31.980887 20643 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:31.957480 22081 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:31.982800 21941 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:31.986179 20467 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:31.986855 22102 consensus_queue.cc:1046] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:31.986871 22056 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:31.975668 22067 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:31.989449 21984 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:31.991606 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 620) took 2071 ms. Trace:
I20250831 13:58:31.941833 21992 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.934468 21932 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.993896 20572 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
W20250831 13:58:31.995026 20616 connection.cc:232] RPC call timeout handler was delayed by -1.0282471369999939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
I20250831 13:58:31.975658 21767 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:31.975939 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:31.987946 22096 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:31.975011 21588 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:31.997334 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:32.021770 20569 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:31.976475 22035 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:31.999171 20494 rpcz_store.cc:276] 0831 13:58:29.920432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.895226 (+1974794us) service_pool.cc:225] Handling call
0831 13:58:31.895532 (+ 306us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.895705 (+ 173us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:31.915088 (+ 19383us) log.cc:844] Serialized 14 byte log entry
0831 13:58:31.990982 (+ 75894us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.991034 (+ 52us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.991068 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:31.991127 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:31.991280 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1002,"prepare.queue_time_us":159,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":40,"raft.queue_time_us":74,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":75520,"threads_started":1}
I20250831 13:58:31.980616 21609 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:31.957747 21983 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:31.990599 21931 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.000185 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.016337 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.017961 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:31.975495 22075 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.019698 20616 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:31.992290 21914 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:31.994316 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.022418 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:32.026727 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:32.028383 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250831 13:58:32.045612 22035 raft_consensus.cc:695] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:32.107879 22070 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.192085 22035 consensus_queue.cc:237] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.117247 22081 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:32.552886 22081 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:32.193365 20644 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
W20250831 13:58:32.144477 22085 log.cc:927] Time spent T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd: Append to log took a long time: real 0.410s user 0.004s sys 0.000s
W20250831 13:58:32.079043 22104 log.cc:927] Time spent T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.106s user 0.001s sys 0.000s
I20250831 13:58:32.193615 22070 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:32.026873 22105 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:32.746158 22105 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.748837 22105 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:32.218971 21988 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:32.751163 20651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 632) took 2983 ms (client timeout 2026 ms). Trace:
I20250831 13:58:32.045421 22096 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.298220 21983 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:32.751461 20400 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54880 (request call id 36) took 6621 ms. Trace:
W20250831 13:58:32.464774 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:32.484984 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:32.485342 20498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 636) took 2463 ms (client timeout 2583 ms). Trace:
I20250831 13:58:32.490135 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:32.752645 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 907) took 3230 ms. Trace:
W20250831 13:58:32.492705 20499 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1055) took 2984 ms (client timeout 2040 ms). Trace:
I20250831 13:58:32.532980 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1045) took 3328 ms. Trace:
I20250831 13:58:32.010419 21565 raft_consensus.cc:2802] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 13:58:32.536193 21173 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:32.542770 22107 log.cc:927] Time spent T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.474s user 0.004s sys 0.012s
W20250831 13:58:32.752477 20498 rpcz_store.cc:269] 0831 13:58:30.022187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.284838 (+262651us) service_pool.cc:225] Handling call
0831 13:58:32.485168 (+2200330us) spinlock_profiling.cc:243] Waited 94.4 ms on lock 0x7b54002dfd54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:32.485270 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":217049344}
I20250831 13:58:32.753281 20502 rpcz_store.cc:276] 0831 13:58:29.204894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.206389 (+ 1495us) service_pool.cc:225] Handling call
0831 13:58:29.206746 (+ 357us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:29.525070 (+318324us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:29.525098 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:29.525133 (+ 35us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:29.525177 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:32.532653 (+3007476us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b5400127fd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:32.532701 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:32.532883 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":731659520}
I20250831 13:58:32.109143 21896 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.755084 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:32.755786 20498 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:32.756167 21896 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.756418 22096 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:32.573305 20568 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.019s sys 0.023s
I20250831 13:58:32.103590 21992 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:32.591495 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 569136us
I20250831 13:58:32.027894 20643 raft_consensus.cc:2953] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
W20250831 13:58:32.757408 20568 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.803s user 0.019s sys 0.024s
I20250831 13:58:32.027231 21588 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:32.759956 21204 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:32.753077 20648 rpcz_store.cc:276] 0831 13:58:29.522445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.397592 (+875147us) service_pool.cc:225] Handling call
0831 13:58:32.751083 (+2353491us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:32.751283 (+ 200us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:32.751312 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:32.751350 (+ 38us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:32.751379 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:32.751440 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:32.752483 (+ 1043us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b54000307d0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:32.752582 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5411905792}
I20250831 13:58:32.761287 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.605070 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1043) took 3400 ms. Trace:
W20250831 13:58:32.617309 20617 connection.cc:232] RPC call timeout handler was delayed by -2.1410309380000285s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16610000000000003s for a response
W20250831 13:58:32.762586 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:32.762475 21202 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:58:32.763408 20617 connection.cc:232] RPC call timeout handler was delayed by -2.04312794599997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
W20250831 13:58:32.763746 20617 connection.cc:232] RPC call timeout handler was delayed by -2.0026907690000257s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1587s for a response
W20250831 13:58:32.764165 20617 connection.cc:232] RPC call timeout handler was delayed by -1.9190398869999967s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20250831 13:58:32.764454 20617 connection.cc:232] RPC call timeout handler was delayed by -1.9099103640000408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1937s for a response
W20250831 13:58:32.764731 20617 connection.cc:232] RPC call timeout handler was delayed by -1.8577103640000132s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1995s for a response
W20250831 13:58:32.764984 20617 connection.cc:232] RPC call timeout handler was delayed by -1.83434204699995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1932s for a response
W20250831 13:58:32.765304 20617 connection.cc:232] RPC call timeout handler was delayed by -1.8118103639999958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
I20250831 13:58:32.765616 22095 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:32.195461 22075 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:32.766307 20499 rpcz_store.cc:269] 0831 13:58:29.508021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.509075 (+ 1054us) service_pool.cc:225] Handling call
0831 13:58:32.492467 (+2983392us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54001c2854. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:32.492634 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":470720384}
I20250831 13:58:32.767771 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.643466 21731 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:32.768882 21731 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.769143 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.644598 21206 raft_consensus.cc:1238] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:32.769891 21992 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:32.766034 21588 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.770418 21992 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.770978 21731 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.646150 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.771682 21588 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:32.772094 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:32.772432 21992 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.772769 20570 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:32.761847 20495 rpcz_store.cc:276] 0831 13:58:29.204250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.204391 (+ 141us) service_pool.cc:225] Handling call
0831 13:58:29.275958 (+ 71567us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:29.525871 (+249913us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:29.525913 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:29.525965 (+ 52us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:29.526006 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:32.604833 (+3078827us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54001865d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:32.604880 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:32.605013 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":574275328}
I20250831 13:58:32.773766 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:32.648686 20571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.007s sys 0.007s
I20250831 13:58:32.774756 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:32.774919 20571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.007s sys 0.007s
I20250831 13:58:32.775758 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.776291 20495 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:32.776247 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.762373 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:32.071856 20569 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:32.777271 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.052794 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:32.651628 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.081878 21932 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.777814 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:32.028614 21609 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.778244 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.657328 20642 raft_consensus.cc:2391] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:32.664601 20541 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:32.675868 21986 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.664258 21659 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:32.678568 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 622) took 2757 ms. Trace:
I20250831 13:58:32.681636 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 954) took 2502 ms. Trace:
I20250831 13:58:32.021942 21767 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:32.684743 20493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37606 (request call id 26) took 3163 ms (client timeout 1579 ms). Trace:
I20250831 13:58:32.684959 21914 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:32.101874 22059 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250831 13:58:32.027966 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:32.693416 22071 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Append to log took a long time: real 1.501s user 0.005s sys 0.000s
I20250831 13:58:32.067307 21985 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:32.697609 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.126902 22069 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.700999 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 972) took 2251 ms. Trace:
I20250831 13:58:32.005155 22056 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.703943 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.704602 20645 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:32.704887 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 685224us
I20250831 13:58:32.705437 21580 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
W20250831 13:58:32.706081 21941 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s user 0.016s sys 0.006s
I20250831 13:58:32.004968 22067 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.011094 21473 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:32.713160 21909 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:32.707799 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.716042 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:32.716197 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:32.716724 21931 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:32.717497 21747 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:32.718513 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1061) took 3208 ms. Trace:
I20250831 13:58:32.714558 20644 raft_consensus.cc:1273] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250831 13:58:32.719173 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.724414 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:58:32.730857 21708 raft_consensus.cc:2953] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Committing config change with OpId 3.5: config changed from index 2 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:32.735275 22100 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:32.735414 22066 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:32.741564 20577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1071) took 3164 ms (client timeout 2314 ms). Trace:
I20250831 13:58:32.744552 21596 raft_consensus.cc:3053] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:32.459808 22078 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:32.751552 20651 rpcz_store.cc:269] 0831 13:58:29.768288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.031952 (+263664us) service_pool.cc:225] Handling call
0831 13:58:32.750957 (+2719005us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54000307d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:32.751081 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":521370496}
I20250831 13:58:32.723600 20397 catalog_manager.cc:5582] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:32.753463 21565 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:32.755132 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.752244 20576 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:32.752940 20646 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:32.591611 22003 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:32.626042 22120 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:32.626608 20501 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 625) took 2704 ms (client timeout 1766 ms). Trace:
I20250831 13:58:32.776638 20498 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
W20250831 13:58:32.765588 20617 connection.cc:232] RPC call timeout handler was delayed by -1.806410364000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
I20250831 13:58:32.801791 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:32.883204 20575 rpcz_store.cc:276] 0831 13:58:30.179793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.176747 (+996954us) service_pool.cc:225] Handling call
0831 13:58:31.177081 (+ 334us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.245998 (+ 68917us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:31.246026 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:31.246064 (+ 38us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:31.246092 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:32.681422 (+1435330us) spinlock_profiling.cc:243] Waited 68.7 ms on lock 0x7b54000b45d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:32.681463 (+ 41us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:32.681591 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":158105856}
W20250831 13:58:32.884627 21941 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s user 0.020s sys 0.009s
I20250831 13:58:32.884769 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.885891 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.886848 21985 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.888924 21985 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.780022 21609 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.751972 20400 rpcz_store.cc:276] 0831 13:58:26.129624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:26.129856 (+ 232us) service_pool.cc:225] Handling call
0831 13:58:32.751376 (+6621520us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:30.483203 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:30.756211 (+273008us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:30.770649 (+ 14438us) write_op.cc:435] Acquired schema lock
0831 13:58:30.770695 (+ 46us) tablet.cc:660] Decoding operations
0831 13:58:31.069712 (+299017us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:31.094464 (+ 24752us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:31.099442 (+ 4978us) tablet.cc:683] Acquiring locks for 1 operations
0831 13:58:31.136768 (+ 37326us) tablet.cc:699] Row locks acquired
0831 13:58:31.136797 (+ 29us) write_op.cc:260] PREPARE: finished
0831 13:58:31.136992 (+ 195us) write_op.cc:270] Start()
0831 13:58:31.137133 (+ 141us) write_op.cc:276] Timestamp: P: 1756648711136935 usec, L: 0
0831 13:58:31.137155 (+ 22us) op_driver.cc:348] REPLICATION: starting
0831 13:58:31.137920 (+ 765us) log.cc:844] Serialized 538 byte log entry
0831 13:58:32.140457 (+1002537us) spinlock_profiling.cc:243] Waited 7.83 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:32.557594 (+417137us) op_driver.cc:464] REPLICATION: finished
0831 13:58:32.558514 (+ 920us) write_op.cc:301] APPLY: starting
0831 13:58:32.558579 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:32.558643 (+ 64us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:32.558658 (+ 15us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:32.558929 (+ 271us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:32.559034 (+ 105us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0831 13:58:32.559062 (+ 28us) write_op.cc:312] APPLY: finished
0831 13:58:32.559270 (+ 208us) log.cc:844] Serialized 22 byte log entry
0831 13:58:32.559416 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:32.559471 (+ 55us) write_op.cc:454] Released schema lock
0831 13:58:32.559497 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:32.559741 (+ 244us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:32.559754 (+ 13us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":439432,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":714,"apply.run_cpu_time_us":1212,"apply.run_wall_time_us":1213,"num_ops":1,"prepare.queue_time_us":982858,"prepare.run_cpu_time_us":71401,"prepare.run_wall_time_us":1124752,"raft.queue_time_us":2416,"raft.run_cpu_time_us":1043,"raft.run_wall_time_us":1197,"replication_time_us":1420076,"spinlock_wait_cycles":18012928,"thread_start_us":1213117,"threads_started":4,"wal-append.queue_time_us":596319}]]}
I20250831 13:58:32.780046 20647 raft_consensus.cc:2408] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 13:58:32.890831 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:32.891073 20500 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:32.891611 20500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1052) took 3466 ms (client timeout 1536 ms). Trace:
I20250831 13:58:32.892128 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.892735 20647 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:32.893810 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.894292 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 621) took 3520 ms. Trace:
I20250831 13:58:32.894006 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.895254 20574 raft_consensus.cc:2391] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:32.895619 21210 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:32.894666 20649 rpcz_store.cc:276] 0831 13:58:29.373930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.042349 (+668419us) service_pool.cc:225] Handling call
0831 13:58:30.077508 (+ 35159us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:30.077842 (+ 334us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:30.077871 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:30.097217 (+ 19346us) log.cc:844] Serialized 26 byte log entry
0831 13:58:30.493932 (+396715us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:30.493967 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:30.494028 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:32.692910 (+2198882us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:32.894013 (+201103us) raft_consensus.cc:1683] finished
0831 13:58:32.894034 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:32.894225 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9086,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":18257,"spinlock_wait_cycles":39701888,"thread_start_us":341013,"threads_started":1,"wal-append.queue_time_us":985353}
I20250831 13:58:32.896338 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.896840 21909 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:32.898880 21747 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.900952 21747 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.901113 21473 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.892346 21931 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.784550 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:32.862921 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 61613.8us
I20250831 13:58:32.879206 20543 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:32.766129 22095 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.894970 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.778705 20569 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:32.777786 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:32.903090 21473 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.854763 22105 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:32.904281 22056 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.921677 22105 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.886317 20497 rpcz_store.cc:276] 0831 13:58:29.509780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.157510 (+1647730us) service_pool.cc:225] Handling call
0831 13:58:32.626576 (+1469066us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:32.626779 (+ 203us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:32.626809 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:32.626849 (+ 40us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:32.626884 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:32.626926 (+ 42us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:32.718321 (+ 91395us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54000118d0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:32.718430 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3356696320}
I20250831 13:58:32.905045 20573 rpcz_store.cc:276] 0831 13:58:30.449563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.539113 (+ 89550us) service_pool.cc:225] Handling call
0831 13:58:32.700850 (+2161737us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54001a63d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:32.700955 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3132254080}
I20250831 13:58:32.923164 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.923784 22105 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.924057 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:32.908995 22067 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.932696 20573 raft_consensus.cc:2366] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6.
I20250831 13:58:32.909184 20571 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:32.933558 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:32.909606 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:32.934816 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.935504 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.909842 21596 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:32.936551 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.936703 21596 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.937100 20467 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:32.937619 21596 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:32.909772 22100 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.910557 21659 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.940667 22100 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:32.940784 21659 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:32.910648 21565 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:32.909381 22003 raft_consensus.cc:695] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:32.942030 20645 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:32.942263 22003 consensus_queue.cc:237] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.943446 21932 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:32.943921 21932 raft_consensus.cc:695] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:32.944763 21932 consensus_queue.cc:237] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.912634 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.913254 22123 consensus_queue.cc:1046] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20250831 13:58:32.947816 22123 raft_consensus.cc:3053] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:32.948139 22123 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:32.948549 20467 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:32.948854 22123 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.950047 22123 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:32.950830 21992 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:32.913394 21960 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:32.910405 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.946451 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.953085 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.953167 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.954373 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.954910 20644 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:32.914891 21941 consensus_queue.cc:1046] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20250831 13:58:32.956626 21941 raft_consensus.cc:3053] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:32.956969 21941 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:32.958061 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:32.891948 20500 rpcz_store.cc:269] 0831 13:58:29.424974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:29.443035 (+ 18061us) service_pool.cc:225] Handling call
0831 13:58:32.703797 (+3260762us) spinlock_profiling.cc:243] Waited 930 us on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:32.891517 (+187720us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2137984}
I20250831 13:58:32.958887 20649 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:32.959370 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.959921 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:32.960332 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.961407 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.915679 22124 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:32.911670 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.905609 20494 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:32.963526 20499 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:32.905288 22066 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.964339 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:32.965274 22066 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.905956 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.966425 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.916568 21207 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:32.919682 21580 raft_consensus.cc:695] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:32.967516 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.968107 20651 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:32.967804 21580 consensus_queue.cc:237] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:32.906203 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 241659us
I20250831 13:58:32.915536 20502 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:32.905017 22120 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.906410 21609 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:32.969313 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 651) took 1036 ms. Trace:
I20250831 13:58:32.907423 22078 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:32.929286 20636 tablet_service.cc:1515] Processing DeleteTablet for tablet 0c14599239fd4c99b3331692a513a092 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:57684
I20250831 13:58:32.908078 21731 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:32.908866 20496 rpcz_store.cc:276] 0831 13:58:29.920988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.929940 (+2008952us) service_pool.cc:225] Handling call
0831 13:58:31.954529 (+ 24589us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.954773 (+ 244us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:31.955032 (+ 259us) log.cc:844] Serialized 14 byte log entry
0831 13:58:32.662386 (+707354us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b1800438fa0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba2da 00007fb9bc9809da
0831 13:58:32.678250 (+ 15864us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:32.678301 (+ 51us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:32.678339 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:32.678402 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:32.678516 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":99,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":1486,"raft.queue_time_us":98,"raft.run_cpu_time_us":76,"raft.run_wall_time_us":77,"spinlock_wait_cycles":3238144,"thread_start_us":15666,"threads_started":1,"wal-append.queue_time_us":722260}
I20250831 13:58:32.977488 21731 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.977840 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:32.979001 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.979314 21731 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:32.908918 20577 rpcz_store.cc:269] 0831 13:58:29.577570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.162013 (+584443us) service_pool.cc:225] Handling call
0831 13:58:32.741341 (+2579328us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54000e6d54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:32.741474 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2118424448}
I20250831 13:58:32.911643 22069 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.908807 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.933591 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.980618 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.981168 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.981608 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:32.905740 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.963444 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 652) took 1027 ms. Trace:
W20250831 13:58:32.905666 20501 rpcz_store.cc:269] 0831 13:58:29.921797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:30.015097 (+ 93300us) service_pool.cc:225] Handling call
0831 13:58:32.626409 (+2611312us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b54000118d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:32.626542 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326766848}
I20250831 13:58:32.982184 20573 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:32.884352 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250831 13:58:32.908970 20493 rpcz_store.cc:269] 0831 13:58:29.521767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:32.684553 (+3162786us) service_pool.cc:212] Skipping call since client already timed out
0831 13:58:32.684668 (+ 115us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 13:58:32.982965 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.919307 20648 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 3
I20250831 13:58:32.983319 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.982460 20494 rpcz_store.cc:276] 0831 13:58:31.935612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:32.027738 (+ 92126us) service_pool.cc:225] Handling call
0831 13:58:32.963390 (+935652us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:32.984200 20493 raft_consensus.cc:2391] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:32.984366 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.985109 20494 raft_consensus.cc:2408] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20250831 13:58:32.969470 21609 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:32.986294 21609 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:32.916349 22095 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:32.920904 20617 connection.cc:232] RPC call timeout handler was delayed by -1.7803103640000018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
I20250831 13:58:32.907315 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:32.985890 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.905956 20616 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:32.987689 20642 raft_consensus.cc:2366] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5.
I20250831 13:58:32.905216 21209 raft_consensus.cc:2953] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
W20250831 13:58:32.988010 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 82137.4us
W20250831 13:58:32.989012 20617 connection.cc:232] RPC call timeout handler was delayed by -1.6971399089999863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
I20250831 13:58:32.989140 20397 catalog_manager.cc:5582] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:32.957732 21941 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:32.972680 22120 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.905424 21914 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.904978 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.907431 21931 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:32.934420 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.975662 21205 raft_consensus.cc:2408] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 13:58:32.992086 21645 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:32.992545 21645 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:32.992861 21645 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:32.993256 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 638) took 1067 ms. Trace:
I20250831 13:58:32.993351 21941 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:32.993609 20495 rpcz_store.cc:276] 0831 13:58:31.926013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:32.905521 (+979508us) service_pool.cc:225] Handling call
0831 13:58:32.993190 (+ 87669us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:32.935521 20571 raft_consensus.cc:2391] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:32.981236 20577 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:32.981834 20650 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:32.987700 20494 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:32.994879 20385 catalog_manager.cc:4928] TS 44d163094add49049cc749547d891602 (127.18.233.131:45067): tablet 0c14599239fd4c99b3331692a513a092 (table test-workload [id=bf9639b2cb72453d8b67d48646dc2938]) successfully deleted
I20250831 13:58:32.995414 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.995661 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.996403 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.996675 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.997044 20650 raft_consensus.cc:2391] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:32.997678 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.979992 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.984983 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:32.989910 20617 connection.cc:232] RPC call timeout handler was delayed by -1.616330937999976s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22440000000000004s for a response
I20250831 13:58:32.914120 20543 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:33.006541 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 92474.5us
I20250831 13:58:33.005030 20569 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:33.007828 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:32.962376 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.990446 20574 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:33.008450 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.988517 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.015642 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:32.971736 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.999662 20501 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:32.970219 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.017150 20651 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:32.920200 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:33.024271 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:32.962347 22124 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:32.924391 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:32.976750 20502 rpcz_store.cc:276] 0831 13:58:31.932732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:32.894738 (+962006us) service_pool.cc:225] Handling call
0831 13:58:32.969210 (+ 74472us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.031584 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:32.937546 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.032668 20498 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:32.998663 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.034263 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.035001 20494 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:33.035917 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.036512 20494 raft_consensus.cc:2374] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5. Re-sending same reply.
I20250831 13:58:33.037418 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.000988 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.040333 20575 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 1.
I20250831 13:58:33.003563 20467 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:32.752892 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:33.041436 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:33.004509 20647 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.042379 20467 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:32.991765 20572 raft_consensus.cc:2391] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:33.009088 20617 connection.cc:232] RPC call timeout handler was delayed by -1.3111741559999928s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
I20250831 13:58:33.010152 20576 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:33.044024 20617 connection.cc:232] RPC call timeout handler was delayed by -1.1804990379999936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2127s for a response
I20250831 13:58:33.044332 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 992) took 1019 ms. Trace:
W20250831 13:58:33.044543 20617 connection.cc:232] RPC call timeout handler was delayed by -1.1782622699999479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2127s for a response
W20250831 13:58:33.045014 20617 connection.cc:232] RPC call timeout handler was delayed by -1.0946463000000222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
I20250831 13:58:33.044879 20576 rpcz_store.cc:276] 0831 13:58:32.024873 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:32.484728 (+459855us) service_pool.cc:225] Handling call
0831 13:58:33.044240 (+559512us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:33.045548 20617 connection.cc:232] RPC call timeout handler was delayed by -1.0121990380000057s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23140000000000002s for a response
I20250831 13:58:32.921770 20570 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:32.994467 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:32.999366 20496 raft_consensus.cc:2366] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6.
I20250831 13:58:33.048964 20543 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:33.050209 20576 raft_consensus.cc:1238] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20250831 13:58:33.051823 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:33.052224 20543 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:33.052891 20466 connection.cc:232] RPC call timeout handler was delayed by -1.7189728219999552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
I20250831 13:58:33.053262 20502 raft_consensus.cc:2408] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:33.021811 20648 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:33.008008 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 985) took 1092 ms. Trace:
I20250831 13:58:33.054921 20572 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 13:58:33.024201 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.056178 20572 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250831 13:58:33.056339 20569 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:33.057382 20572 raft_consensus.cc:1238] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:33.058566 20569 raft_consensus.cc:1238] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:33.005676 21715 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:33.059401 21715 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:33.053411 20466 connection.cc:232] RPC call timeout handler was delayed by -1.3714431849999755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24500000000000002s for a response
I20250831 13:58:33.054643 20568 rpcz_store.cc:276] 0831 13:58:31.915788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:31.933349 (+ 17561us) service_pool.cc:225] Handling call
0831 13:58:31.933640 (+ 291us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:31.933847 (+ 207us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:32.757946 (+824099us) log.cc:844] Serialized 14 byte log entry
0831 13:58:33.007162 (+249216us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.007208 (+ 46us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:33.007792 (+ 584us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.007859 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.007960 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":611,"prepare.queue_time_us":40339,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":1663,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":10766,"thread_start_us":1052566,"threads_started":2,"wal-append.queue_time_us":177001}
I20250831 13:58:33.063251 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.064035 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.065164 20467 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.065843 20497 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:33.065948 20569 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:32.982450 21767 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.027026 21208 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:33.077562 20502 raft_consensus.cc:1273] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:33.078161 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 632) took 3120 ms. Trace:
I20250831 13:58:33.078440 20502 rpcz_store.cc:276] 0831 13:58:29.957582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.070462 (+3112880us) service_pool.cc:225] Handling call
0831 13:58:33.071058 (+ 596us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.077961 (+ 6903us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.078116 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.026574 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.021905 20397 catalog_manager.cc:5582] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:33.079793 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 637) took 3057 ms. Trace:
I20250831 13:58:33.028828 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:33.029695 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 60500us
I20250831 13:58:33.080114 20502 rpcz_store.cc:276] 0831 13:58:30.022482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.079113 (+3056631us) service_pool.cc:225] Handling call
0831 13:58:33.079338 (+ 225us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.079492 (+ 154us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:33.079522 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.079559 (+ 37us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:33.079587 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.079645 (+ 58us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.079751 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.080879 20541 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:33.023754 22066 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.080008 20397 catalog_manager.cc:5582] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:33.086373 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1071) took 2871 ms. Trace:
I20250831 13:58:33.025486 20499 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:33.023896 21174 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:33.025673 20398 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59942 (request call id 146) took 7330 ms. Trace:
I20250831 13:58:33.066363 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 630) took 3109 ms. Trace:
I20250831 13:58:33.066196 20467 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.067068 20543 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.068751 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 650) took 1136 ms. Trace:
I20250831 13:58:33.069453 20466 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.076844 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.077054 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.081712 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:33.030230 20644 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:33.082319 20576 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:33.029158 21588 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:33.027586 21565 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.084342 20574 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:33.086833 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 624) took 3165 ms. Trace:
I20250831 13:58:33.083199 20649 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.210677 20495 rpcz_store.cc:276] 0831 13:58:29.921532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.068518 (+3146986us) service_pool.cc:225] Handling call
0831 13:58:33.068760 (+ 242us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.081901 (+ 13141us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:33.081937 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.081976 (+ 39us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:33.082003 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.086613 (+ 4610us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54001621d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:33.086651 (+ 38us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.086784 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29847296}
I20250831 13:58:33.085843 20397 catalog_manager.cc:5582] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 2 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:33.210964 20575 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:33.213109 20397 catalog_manager.cc:5582] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 2, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 2 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:33.214946 20500 raft_consensus.cc:2408] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:33.116851 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 640) took 1190 ms. Trace:
I20250831 13:58:33.216116 20501 rpcz_store.cc:276] 0831 13:58:31.926582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.090541 (+1163959us) service_pool.cc:225] Handling call
0831 13:58:33.116255 (+ 25714us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.116460 (+ 205us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:33.116494 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.116550 (+ 56us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:33.116594 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.116664 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.116787 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.087919 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 639) took 1161 ms. Trace:
I20250831 13:58:33.217350 20499 rpcz_store.cc:276] 0831 13:58:31.926308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.087091 (+1160783us) service_pool.cc:225] Handling call
0831 13:58:33.087349 (+ 258us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.087572 (+ 223us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:33.087606 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.087656 (+ 50us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:33.087698 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.087766 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.087874 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.125308 20647 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:33.088485 20645 raft_consensus.cc:1238] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 13:58:33.209873 20497 rpcz_store.cc:276] 0831 13:58:29.956874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.065364 (+3108490us) service_pool.cc:225] Handling call
0831 13:58:33.065607 (+ 243us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.066173 (+ 566us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.066320 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.231693 20495 raft_consensus.cc:1273] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250831 13:58:33.232057 20495 pending_rounds.cc:85] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 1
I20250831 13:58:33.232277 20495 pending_rounds.cc:107] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250831 13:58:33.232451 20644 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
W20250831 13:58:33.221848 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 137038us
I20250831 13:58:33.222045 20573 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
W20250831 13:58:33.222821 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 155833us
I20250831 13:58:33.076491 21747 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:33.221104 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 151769us
I20250831 13:58:33.232479 20495 raft_consensus.cc:2887] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:33.086725 20502 rpcz_store.cc:276] 0831 13:58:30.214826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.085604 (+2870778us) service_pool.cc:225] Handling call
0831 13:58:33.085843 (+ 239us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.085986 (+ 143us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:33.086009 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.086045 (+ 36us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:33.086166 (+ 121us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.086226 (+ 60us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.086329 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:33.222465 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 140828us
I20250831 13:58:33.239434 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 641) took 1308 ms. Trace:
I20250831 13:58:33.068586 21960 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:33.239985 20495 rpcz_store.cc:276] 0831 13:58:31.930748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.231113 (+1300365us) service_pool.cc:225] Handling call
0831 13:58:33.231384 (+ 271us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.239165 (+ 7781us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.239363 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:33.241113 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 175014us
I20250831 13:58:33.242846 20651 raft_consensus.cc:1238] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:33.239981 20541 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:33.244741 20497 raft_consensus.cc:1273] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:33.084106 21932 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:33.077605 22100 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 13:58:33.200301 20431 debug-util.cc:398] Leaking SignalData structure 0x7b080018c440 after lost signal to thread 19369
I20250831 13:58:33.085270 22095 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:33.083750 21992 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:33.200891 20642 raft_consensus.cc:1273] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20250831 13:58:33.190104 19371 debug-util.cc:398] Leaking SignalData structure 0x7b080000ffa0 after lost signal to thread 20616
I20250831 13:58:33.282073 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 647) took 1350 ms. Trace:
I20250831 13:58:33.248426 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52966 (request call id 409) took 3596 ms. Trace:
I20250831 13:58:33.091692 22067 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.084724 22066 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:33.252689 22003 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:33.195397 21914 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.234773 20647 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:33.218904 20498 rpcz_store.cc:276] 0831 13:58:31.932245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.063006 (+1130761us) service_pool.cc:225] Handling call
0831 13:58:33.068701 (+ 5695us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.199834 22124 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.107250 22127 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.219276 21731 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:33.242290 22096 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:33.241837 22105 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20250831 13:58:33.300586 21173 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:33.246443 21747 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.282523 20501 rpcz_store.cc:276] 0831 13:58:31.931803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.234885 (+1303082us) service_pool.cc:225] Handling call
0831 13:58:33.281203 (+ 46318us) spinlock_profiling.cc:243] Waited 4.42 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:33.281426 (+ 223us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.281676 (+ 250us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:33.281712 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.281764 (+ 52us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:33.281805 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.281872 (+ 67us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.282001 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10173312}
W20250831 13:58:33.271265 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 271112us
I20250831 13:58:33.242504 21473 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.284694 20617 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.283004 20502 rpcz_store.cc:276] 0831 13:58:29.651978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.247420 (+3595442us) service_pool.cc:225] Handling call
0831 13:58:33.248372 (+ 952us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.202152 21767 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:33.297490 20649 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:33.257118 22120 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:33.246856 21960 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:33.202287 21645 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.209060 21909 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:33.246698 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 241107us
I20250831 13:58:33.257580 22100 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:33.310881 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:33.251966 21985 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20250831 13:58:33.318447 21473 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:33.287196 22067 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:33.288771 21715 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:33.298309 22066 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.298532 22003 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:33.324288 20617 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:33.329200 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 76234.7us
I20250831 13:58:33.312841 22096 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:33.341547 20648 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 13:58:33.218271 20398 rpcz_store.cc:276] 0831 13:58:25.695426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:25.695625 (+ 199us) service_pool.cc:225] Handling call
0831 13:58:33.025623 (+7329998us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:26.600333 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:26.601219 (+ 886us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:26.601279 (+ 60us) write_op.cc:435] Acquired schema lock
0831 13:58:26.601333 (+ 54us) tablet.cc:660] Decoding operations
0831 13:58:26.602871 (+ 1538us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:26.603007 (+ 136us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:26.603048 (+ 41us) tablet.cc:683] Acquiring locks for 11 operations
0831 13:58:26.604583 (+ 1535us) tablet.cc:699] Row locks acquired
0831 13:58:26.604642 (+ 59us) write_op.cc:260] PREPARE: finished
0831 13:58:26.604838 (+ 196us) write_op.cc:270] Start()
0831 13:58:26.605026 (+ 188us) write_op.cc:276] Timestamp: P: 1756648706604762 usec, L: 0
0831 13:58:26.605071 (+ 45us) op_driver.cc:348] REPLICATION: starting
0831 13:58:26.605974 (+ 903us) log.cc:844] Serialized 4450 byte log entry
0831 13:58:27.558028 (+952054us) op_driver.cc:464] REPLICATION: finished
0831 13:58:28.060155 (+502127us) write_op.cc:301] APPLY: starting
0831 13:58:28.060255 (+ 100us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:28.060496 (+ 241us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:28.060632 (+ 136us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:28.329435 (+268803us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:28.373092 (+ 43657us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0831 13:58:28.384034 (+ 10942us) write_op.cc:312] APPLY: finished
0831 13:58:28.671635 (+287601us) log.cc:844] Serialized 82 byte log entry
0831 13:58:28.723894 (+ 52259us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:28.724063 (+ 169us) write_op.cc:454] Released schema lock
0831 13:58:28.724145 (+ 82us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:28.771619 (+ 47474us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:28.772162 (+ 543us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":80273,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":464124,"apply.run_cpu_time_us":11453,"apply.run_wall_time_us":670863,"num_ops":11,"prepare.queue_time_us":153021,"prepare.run_cpu_time_us":17369,"prepare.run_wall_time_us":59326,"raft.queue_time_us":383525,"raft.run_cpu_time_us":29255,"raft.run_wall_time_us":322750,"replication_time_us":888901,"thread_start_us":433941,"threads_started":5,"wal-append.queue_time_us":281678,"wal-append.run_cpu_time_us":15187,"wal-append.run_wall_time_us":2527588}]]}
W20250831 13:58:33.322957 20431 debug-util.cc:398] Leaking SignalData structure 0x7b0800035020 after lost signal to thread 20434
I20250831 13:58:33.311108 22105 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:33.364452 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 120872us
W20250831 13:58:33.365432 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 125560us
W20250831 13:58:33.300514 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 374245us
I20250831 13:58:33.399149 22003 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:33.403322 21715 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:33.409408 21985 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:33.390642 22067 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.279188 20500 raft_consensus.cc:1273] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:33.321092 22120 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:33.383628 21747 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.383697 21645 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:33.323673 21960 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:33.336632 21659 raft_consensus.cc:2762] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 13:58:33.337986 22100 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:33.395529 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:33.396790 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 72580.5us
I20250831 13:58:33.452775 20617 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.313040 21731 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:33.397006 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
W20250831 13:58:33.398695 20646 consensus_meta.cc:287] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.501s user 0.004s sys 0.014s
I20250831 13:58:33.410033 22105 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:33.385248 21473 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.319702 21914 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.390739 21932 raft_consensus.cc:695] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:33.403699 22096 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:33.394094 21208 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:33.336597 21896 consensus_queue.cc:1046] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20250831 13:58:33.477869 20617 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:33.494166 20617 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:58:33.477355 22133 log.cc:927] Time spent T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
W20250831 13:58:33.482018 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 75273.7us
I20250831 13:58:33.447836 22120 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:33.485296 21914 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:33.487437 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 623) took 3566 ms. Trace:
I20250831 13:58:33.485232 21983 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:33.453960 21565 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:33.490604 22078 consensus_meta.cc:287] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.514s user 0.001s sys 0.008s
W20250831 13:58:33.455073 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 52981.7us
I20250831 13:58:33.447913 21174 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:33.478945 20646 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:33.520401 22138 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20250831 13:58:33.520763 22069 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader election won for term 4
W20250831 13:58:33.521719 22137 log.cc:927] Time spent T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20250831 13:58:33.521946 20643 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 13:58:33.521752 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 990) took 1497 ms. Trace:
I20250831 13:58:33.479800 21992 raft_consensus.cc:695] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:33.525307 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 658) took 1530 ms. Trace:
W20250831 13:58:33.526473 22141 log.cc:927] Time spent T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20250831 13:58:33.525664 20650 rpcz_store.cc:276] 0831 13:58:31.994982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:32.997559 (+1002577us) service_pool.cc:225] Handling call
0831 13:58:32.997850 (+ 291us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:32.998226 (+ 376us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:32.998259 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:32.998809 (+ 550us) log.cc:844] Serialized 52 byte log entry
0831 13:58:33.156677 (+157868us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:33.240841 (+ 84164us) log.cc:844] Serialized 14 byte log entry
0831 13:58:33.260124 (+ 19283us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b1800396580. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b998c65a 00007fb9c2e5d409 00007fb9b984daf0 00007fb9b9a11894 00007fb9b9a11805 00007fb9bc96e020 00007fb9bc9806c6 00007fb9bc991237 00007fb9bc9911b8
0831 13:58:33.260315 (+ 191us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.260403 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:33.524983 (+264580us) raft_consensus.cc:1683] finished
0831 13:58:33.525093 (+ 110us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.525254 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8663,"prepare.queue_time_us":84149,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":9584,"raft.queue_time_us":20004,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":8804,"spinlock_wait_cycles":12237184,"thread_start_us":157451,"threads_started":1,"wal-append.queue_time_us":313880}
I20250831 13:58:33.528151 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1077) took 1510 ms. Trace:
W20250831 13:58:33.512231 21609 consensus_meta.cc:287] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.525s user 0.007s sys 0.002s
I20250831 13:58:33.528523 20577 rpcz_store.cc:276] 0831 13:58:32.017505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.016622 (+999117us) service_pool.cc:225] Handling call
0831 13:58:33.016886 (+ 264us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:33.017216 (+ 330us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:33.017245 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:33.017652 (+ 407us) log.cc:844] Serialized 26 byte log entry
0831 13:58:33.262091 (+244439us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:33.384446 (+122355us) log.cc:844] Serialized 14 byte log entry
0831 13:58:33.390785 (+ 6339us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.390856 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:33.527969 (+137113us) raft_consensus.cc:1683] finished
0831 13:58:33.527988 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.528101 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":134166,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":88,"raft.queue_time_us":965,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":1775,"thread_start_us":247800,"threads_started":2,"wal-append.queue_time_us":249873}
I20250831 13:58:33.526015 21992 consensus_queue.cc:237] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.513671 21914 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.498183 21659 raft_consensus.cc:2762] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20250831 13:58:33.538759 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 133172us
I20250831 13:58:33.541783 20467 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:58:33.550485 20616 leader_election.cc:336] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629): Timed out: RequestConsensusVote RPC to 127.18.233.130:44629 timed out after 1.536s (SENT)
I20250831 13:58:33.496253 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:33.451134 20493 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:33.515952 22142 log.cc:927] Time spent T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250831 13:58:33.488957 22149 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:33.519886 22140 log.cc:927] Time spent T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.138s user 0.000s sys 0.000s
W20250831 13:58:33.520002 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 109914us
I20250831 13:58:33.520860 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 655) took 1527 ms. Trace:
I20250831 13:58:33.499310 21715 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:33.522894 20571 rpcz_store.cc:276] 0831 13:58:32.024235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.018521 (+994286us) service_pool.cc:225] Handling call
0831 13:58:33.026888 (+ 8367us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:33.027201 (+ 313us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:33.027230 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:33.027638 (+ 408us) log.cc:844] Serialized 26 byte log entry
0831 13:58:33.223561 (+195923us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:33.223597 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.223672 (+ 75us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:33.521534 (+297862us) raft_consensus.cc:1683] finished
0831 13:58:33.521566 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.521699 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":271,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":195557,"threads_started":1,"wal-append.queue_time_us":272590}
I20250831 13:58:33.486781 21747 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:33.495970 22147 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:33.522998 22078 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:33.556800 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 60645.5us
I20250831 13:58:33.458600 22095 raft_consensus.cc:695] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:33.528577 21941 consensus_meta.cc:287] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.535s user 0.008s sys 0.000s
I20250831 13:58:33.517100 21983 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:33.560963 21588 raft_consensus.cc:695] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:33.565268 21645 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:33.561827 21909 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.577697 20617 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:33.585758 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44083.6us
I20250831 13:58:33.578148 21932 consensus_queue.cc:237] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.577178 22147 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.587334 22155 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:33.589244 20466 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.580447 21747 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.577494 22152 consensus_queue.cc:1046] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250831 13:58:33.597561 22127 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:33.600863 21596 consensus_meta.cc:287] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.663s user 0.007s sys 0.000s
I20250831 13:58:33.601656 22066 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.638280 22147 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.608551 20570 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:33.611023 20467 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.612496 20617 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:58:33.661242 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 48807.3us
I20250831 13:58:33.662235 20617 leader_election.cc:400] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4.
I20250831 13:58:33.546955 21609 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.615662 20466 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:33.616493 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1067) took 3691 ms. Trace:
I20250831 13:58:33.664312 20496 rpcz_store.cc:276] 0831 13:58:29.925514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.067736 (+3142222us) service_pool.cc:225] Handling call
0831 13:58:33.068013 (+ 277us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.068213 (+ 200us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:33.538643 (+470430us) log.cc:844] Serialized 14 byte log entry
0831 13:58:33.616173 (+ 77530us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.616235 (+ 62us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:33.616276 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.616335 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.616443 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19550,"prepare.queue_time_us":122720,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":84,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":2102,"thread_start_us":508799,"threads_started":2}
I20250831 13:58:33.607120 21983 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:33.573660 20646 rpcz_store.cc:276] 0831 13:58:31.993538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:32.489862 (+496324us) service_pool.cc:225] Handling call
0831 13:58:33.520781 (+1030919us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:33.634132 22095 consensus_queue.cc:237] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.618667 21909 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.634517 22127 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.636673 22152 raft_consensus.cc:3053] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:33.638371 22066 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.638546 22096 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:33.640241 22135 log.cc:927] Time spent T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.352s user 0.000s sys 0.002s
I20250831 13:58:33.612890 22157 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:33.613631 21715 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:33.642988 20431 debug-util.cc:398] Leaking SignalData structure 0x7b0800394900 after lost signal to thread 20526
I20250831 13:58:33.643263 21747 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.616348 22155 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:33.670539 21588 consensus_queue.cc:237] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.671880 22144 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:33.679519 22144 raft_consensus.cc:695] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:33.679896 21715 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:33.680615 21715 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:33.680686 22144 consensus_queue.cc:237] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.644080 21731 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:33.574437 22149 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.644393 21985 raft_consensus.cc:3053] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:33.685125 21985 raft_consensus.cc:738] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:33.686651 22149 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.686184 21985 consensus_queue.cc:260] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:33.672961 21909 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:33.644850 21914 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:33.607822 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 107597us
I20250831 13:58:33.688936 20541 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:33.645865 22003 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.646813 22105 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.618731 21960 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.648566 22067 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.513360 20494 rpcz_store.cc:276] 0831 13:58:29.921249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.064212 (+3142963us) service_pool.cc:225] Handling call
0831 13:58:33.064447 (+ 235us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.064606 (+ 159us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:33.104580 (+ 39974us) log.cc:844] Serialized 14 byte log entry
0831 13:58:33.486984 (+382404us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:33.487168 (+ 184us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:33.487210 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.487272 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:33.487378 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6589,"prepare.queue_time_us":42300,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":18305,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":12087,"thread_start_us":374615,"threads_started":1}
I20250831 13:58:33.613219 22120 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:33.648674 21473 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.662634 20617 leader_election.cc:403] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:33.673425 22152 raft_consensus.cc:738] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:33.697074 20617 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.698056 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:33.666644 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 51065.7us
I20250831 13:58:33.669744 22069 raft_consensus.cc:695] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:33.617435 22078 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.612924 22162 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:33.671464 21176 leader_election.cc:336] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.579s (SENT)
I20250831 13:58:33.671676 22123 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:33.670969 21645 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.645782 22100 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.690953 20541 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:33.696887 22075 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4.
I20250831 13:58:33.712241 21914 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.712688 22120 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.713173 22067 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.704320 22080 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:33.713898 22080 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:33.714215 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:33.714388 22066 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:33.714830 20646 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 6
I20250831 13:58:33.715634 21473 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.688247 21909 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.717253 22100 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.714226 22080 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:33.723613 21960 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.723737 20646 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:58:33.723878 22152 consensus_queue.cc:260] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.724782 22003 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.724951 21174 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:58:33.675623 20431 debug-util.cc:398] Leaking SignalData structure 0x7b08003949c0 after lost signal to thread 20527
I20250831 13:58:33.703671 21767 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:33.725855 22120 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader election won for term 6
W20250831 13:58:33.726142 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 55718.5us
W20250831 13:58:33.665865 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 54922.5us
I20250831 13:58:33.723656 22105 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.699167 20466 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:33.675947 22155 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:33.676067 22096 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.673796 21609 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.692724 21731 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:33.739012 21731 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:33.739463 21731 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:33.702191 20617 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.705514 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 991) took 1681 ms. Trace:
W20250831 13:58:33.733949 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:33.740973 20570 rpcz_store.cc:276] 0831 13:58:32.024567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.046546 (+1021979us) service_pool.cc:225] Handling call
0831 13:58:33.046918 (+ 372us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:33.660178 (+613260us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:33.705346 (+ 45168us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b54002fe254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:33.705471 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1291268480}
W20250831 13:58:33.703135 21173 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:33.746089 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.746608 20576 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:33.744963 20569 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:33.687855 21985 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.751937 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.752957 22159 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:33.753628 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.725126 22152 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:33.726382 22120 raft_consensus.cc:695] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 6 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
W20250831 13:58:33.693372 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 296467us
I20250831 13:58:33.759126 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.704398 21645 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:33.738517 20577 raft_consensus.cc:1238] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:33.761888 20541 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:33.741884 20466 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:33.742182 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.742662 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.763669 20576 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:33.763988 20573 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:33.744290 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.744946 22123 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:33.745121 21767 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.757859 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:33.726465 20431 debug-util.cc:398] Leaking SignalData structure 0x7b080038e780 after lost signal to thread 20601
I20250831 13:58:33.778327 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.744290 20617 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:33.759720 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:33.760457 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.760635 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.761003 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.779918 20574 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:33.779892 20501 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:33.761129 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.765702 20466 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.780851 20500 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:33.781388 20466 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:33.781989 20617 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.767666 20648 raft_consensus.cc:1273] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:33.767556 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.768424 20497 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:33.782418 20466 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.768455 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.769749 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.783809 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.784565 20650 raft_consensus.cc:2391] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:33.771422 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.785061 20651 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:33.770970 20616 leader_election.cc:400] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6.
I20250831 13:58:33.784495 20647 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:33.785593 20616 leader_election.cc:403] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250831 13:58:33.775885 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.786621 20466 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:33.776189 21645 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.760898 22120 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:33.789016 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.790930 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.791358 20644 raft_consensus.cc:2391] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:33.791970 21715 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:33.792251 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.791997 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.792704 21932 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:33.792866 20493 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:33.762827 21909 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.793460 21747 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:33.793833 22105 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:33.793931 21747 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:33.794404 22105 raft_consensus.cc:695] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:33.794546 21747 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:33.794909 21909 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.742341 22080 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.795365 22105 consensus_queue.cc:237] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.797317 22080 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:33.750331 21731 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.801234 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.801764 20650 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:33.803071 20466 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:33.805078 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:33.807088 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.807565 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.807749 20495 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.808498 20650 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:33.811256 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:33.787431 20431 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6880 after lost signal to thread 20602
I20250831 13:58:33.778064 21767 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:33.776073 22149 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:33.822118 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.814201 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.811822 20497 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.823861 20649 raft_consensus.cc:2408] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:33.811431 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:33.812551 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.826139 20650 raft_consensus.cc:2391] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:33.826205 20498 raft_consensus.cc:2391] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:33.800706 21596 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:33.817651 20466 leader_election.cc:400] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5.
I20250831 13:58:33.818374 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.832122 20466 leader_election.cc:403] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250831 13:58:33.819036 22148 consensus_queue.cc:1035] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:33.819921 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.811301 22158 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:33.783921 22075 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:33.845247 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.821270 20575 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:33.846544 20572 raft_consensus.cc:1238] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:33.847383 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.848102 20572 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:33.849606 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.828121 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.848976 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.814067 20616 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:33.850615 20498 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:33.835980 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.840849 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:33.810995 21988 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:33.811370 22100 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:33.851716 20573 raft_consensus.cc:2391] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:33.855401 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.821106 21767 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:33.861470 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 45664.9us
I20250831 13:58:33.862411 20466 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:33.862815 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.856086 20573 raft_consensus.cc:2391] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:33.848937 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.863677 20569 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:58:33.864816 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.763120 22127 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.705750 22162 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.837306 21580 consensus_queue.cc:1046] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
I20250831 13:58:33.856596 21988 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:33.859009 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:33.877647 21209 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:33.859755 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.860028 22105 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:33.879154 20572 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:33.879381 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:33.817492 22080 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6.
I20250831 13:58:33.842005 22158 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:33.870366 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.871215 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.899938 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:33.900507 20644 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.900704 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.869372 20495 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:33.873270 20466 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:33.902052 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.871289 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.837656 21596 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:33.876608 20576 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:33.888172 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:33.870357 21932 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:33.816659 21715 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20250831 13:58:33.925369 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.926002 20571 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:33.888334 21988 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:33.930938 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.931514 20575 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:33.897059 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:33.932313 20650 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.934259 20467 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.856882 20646 raft_consensus.cc:1273] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:33.904351 22067 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250831 13:58:33.934846 22157 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:33.935110 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.826517 22069 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:33.896624 22105 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.936482 22003 consensus_queue.cc:1046] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:33.898093 22078 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20250831 13:58:33.937908 22003 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:33.874378 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.887965 21909 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:33.943497 20644 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:33.903797 20573 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 5.
I20250831 13:58:33.910387 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:33.913774 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.906318 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:33.945155 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.945585 20543 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:33.945873 20573 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:33.916121 20498 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:33.918722 20496 raft_consensus.cc:2391] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:33.895450 20431 debug-util.cc:398] Leaking SignalData structure 0x7b0800323c60 after lost signal to thread 20675
W20250831 13:58:33.920142 22170 log.cc:927] Time spent T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
W20250831 13:58:33.920789 22175 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
W20250831 13:58:33.923662 22167 log.cc:927] Time spent T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd: Append to log took a long time: real 0.183s user 0.001s sys 0.000s
I20250831 13:58:33.926131 20568 raft_consensus.cc:2391] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:33.860463 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:33.875798 21992 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5.
I20250831 13:58:33.919102 21596 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:33.917389 22095 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:33.982594 20573 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:58:33.903465 20466 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:33.941471 20574 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
W20250831 13:58:33.894253 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 80264.4us
I20250831 13:58:33.904754 22158 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:33.955643 20576 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:33.956218 20650 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:33.899204 21580 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:33.939553 20577 raft_consensus.cc:1238] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:33.971889 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:33.930503 21747 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:33.982072 20570 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:33.995992 22148 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:33.956877 21896 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.946724 20543 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:33.986865 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:33.895896 21960 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20250831 13:58:33.997985 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:34.004248 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 57612.7us
W20250831 13:58:34.005271 22171 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20250831 13:58:34.006134 22146 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:34.007042 22147 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:34.007390 22162 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.009078 21731 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:33.983563 21983 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.009958 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.011112 20648 raft_consensus.cc:1238] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:34.011023 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.011941 21983 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:34.012220 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:33.995815 21659 consensus_queue.cc:1046] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
W20250831 13:58:34.015046 20431 debug-util.cc:398] Leaking SignalData structure 0x7b0800399dc0 after lost signal to thread 21234
I20250831 13:58:33.991178 21932 consensus_queue.cc:1046] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250831 13:58:34.022365 21580 consensus_queue.cc:1046] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:34.023176 21932 raft_consensus.cc:3053] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:34.023552 21932 raft_consensus.cc:738] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:34.001551 21896 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:34.024384 21896 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:33.986335 20497 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:33.992547 20575 raft_consensus.cc:1273] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:33.991372 20467 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.956013 22080 raft_consensus.cc:3053] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:33.956878 21609 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:33.985263 20573 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:33.975620 22125 raft_consensus.cc:2802] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
W20250831 13:58:34.000883 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 97463.4us
I20250831 13:58:33.950541 22149 consensus_queue.cc:1046] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20250831 13:58:34.005133 20543 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:34.033841 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:33.957064 22123 raft_consensus.cc:3053] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:33.966840 22157 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:34.029714 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.036213 20645 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:34.031219 20647 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:34.024309 21932 consensus_queue.cc:260] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.039665 22100 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:33.975531 21175 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:34.079813 20543 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:34.037425 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 46112.4us
I20250831 13:58:33.999759 21715 consensus_queue.cc:1046] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20250831 13:58:34.030479 22080 raft_consensus.cc:738] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:34.032683 21609 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:34.088871 20645 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:34.036959 22123 raft_consensus.cc:738] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:34.049525 21473 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:34.076370 22100 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:34.077117 22178 raft_consensus.cc:3053] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:34.078439 22003 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
W20250831 13:58:34.104049 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 117893us
I20250831 13:58:34.001221 22148 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.032362 21708 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20250831 13:58:34.106971 20543 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:34.093052 21609 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:34.043318 22125 raft_consensus.cc:695] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:34.108536 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 286637us
I20250831 13:58:34.079319 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.095158 22120 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:34.101063 22078 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:34.082878 21941 consensus_queue.cc:1046] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
W20250831 13:58:34.108474 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 145551us
I20250831 13:58:34.118666 20543 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:34.079422 22157 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20250831 13:58:34.103490 21473 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:34.104818 21932 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:34.109422 22100 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:34.109237 22178 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:34.139444 22173 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20250831 13:58:34.108213 21985 consensus_queue.cc:1046] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20250831 13:58:34.123240 21960 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:34.159917 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.116644 22127 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.115453 22144 consensus_queue.cc:1046] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250831 13:58:34.179528 20543 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:58:34.166567 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 53885.5us
I20250831 13:58:34.131166 22078 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:34.136250 22146 consensus_queue.cc:1046] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20250831 13:58:34.135710 21983 raft_consensus.cc:3053] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:34.137391 22184 consensus_queue.cc:1046] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
W20250831 13:58:34.172549 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 143733us
I20250831 13:58:34.190570 20541 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:34.172822 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 52197.9us
W20250831 13:58:34.175731 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 71250.3us
I20250831 13:58:34.148321 22162 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election won for term 2
W20250831 13:58:34.192445 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 75137.3us
I20250831 13:58:34.137768 20642 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:34.153970 21941 raft_consensus.cc:3053] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:34.148629 21473 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:34.131383 22120 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:34.185117 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.141414 21767 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.191388 20617 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.195250 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:34.220325 20570 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:34.188881 21708 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:34.137524 22123 consensus_queue.cc:260] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.192466 21747 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.129926 22080 consensus_queue.cc:260] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.187103 21909 consensus_queue.cc:1035] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20250831 13:58:34.202658 21731 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:34.187217 21985 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:34.180478 22105 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.221443 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:34.294680 20571 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:34.294512 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.229470 21708 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:34.188817 21983 raft_consensus.cc:738] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:34.228120 22177 log.cc:927] Time spent T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.132s user 0.001s sys 0.000s
I20250831 13:58:34.184487 22149 raft_consensus.cc:3053] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:34.222455 21175 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:34.192756 21960 raft_consensus.cc:695] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:34.246040 20643 raft_consensus.cc:1238] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
W20250831 13:58:34.246223 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:34.219079 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:34.195843 21941 raft_consensus.cc:738] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:34.201593 22120 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:34.257310 20570 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:34.205014 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.186990 22078 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:34.276335 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.277954 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:34.278995 22188 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:34.193428 22125 consensus_queue.cc:237] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:34.287940 22179 log.cc:927] Time spent T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
W20250831 13:58:34.290994 22174 log.cc:927] Time spent T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.285s user 0.001s sys 0.003s
I20250831 13:58:34.290815 21988 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.295356 20496 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:34.223134 20617 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:34.242563 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 48987.7us
I20250831 13:58:34.364284 20649 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:34.364818 20616 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:34.364921 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 141901us
I20250831 13:58:34.246061 21931 consensus_queue.cc:1046] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20250831 13:58:34.366704 20616 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:34.301090 21708 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:34.314504 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.317924 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.323115 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.323225 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.324330 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.326160 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.300922 22003 raft_consensus.cc:695] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:34.327446 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.371497 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:34.333796 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 143285us
I20250831 13:58:34.373071 20541 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.302994 22189 raft_consensus.cc:3053] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:34.328403 20642 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 13:58:34.296008 22184 consensus_queue.cc:1046] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20250831 13:58:34.295912 21985 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:34.307595 22148 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.302983 22149 raft_consensus.cc:738] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:34.370370 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.231470 22178 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:34.316682 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 122832us
I20250831 13:58:34.378921 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.393572 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.315263 22069 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:34.380244 20541 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.400748 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.401028 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.401341 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.401762 20645 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:34.372220 20648 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:34.378100 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.402822 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.404554 20541 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.365572 21983 consensus_queue.cc:260] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.390549 20543 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:34.391904 22184 raft_consensus.cc:3053] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:34.307673 21896 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.378086 22188 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:34.390362 20498 raft_consensus.cc:2953] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 2 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:34.405369 20645 raft_consensus.cc:2391] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:34.396409 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.407030 20617 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.378737 21580 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:34.360332 20493 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:34.411983 20497 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:34.403522 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.419853 20647 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 13:58:34.430994 21983 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:34.432022 22205 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:34.410833 20541 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.378230 22189 raft_consensus.cc:738] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:34.367688 22162 raft_consensus.cc:695] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:34.347538 21473 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.388188 22067 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:34.426944 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.438460 20499 raft_consensus.cc:2408] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:34.406788 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:34.406389 22190 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:34.410897 22105 raft_consensus.cc:2762] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250831 13:58:34.410843 22184 raft_consensus.cc:738] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:34.419123 22148 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:34.516209 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.408653 21715 consensus_queue.cc:1046] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20250831 13:58:34.435856 22192 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:34.521857 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 111202us
I20250831 13:58:34.427757 22178 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:34.426692 21988 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:34.395467 21176 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.450109 21767 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:34.427052 22199 consensus_queue.cc:1046] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.117s
I20250831 13:58:34.392529 21941 consensus_queue.cc:260] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:34.527709 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:34.517058 20649 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
W20250831 13:58:34.520087 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 88909.4us
W20250831 13:58:34.563305 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 156340us
I20250831 13:58:34.518813 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:34.569036 20650 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
W20250831 13:58:34.515426 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 104425us
W20250831 13:58:34.564553 21596 consensus_meta.cc:287] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.574s user 0.010s sys 0.007s
I20250831 13:58:34.488205 22190 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:34.570852 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:58:34.616472 20617 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.617630 20541 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.521061 20646 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:34.516707 22200 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:34.573758 21565 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:34.638767 20617 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.425618 21960 consensus_queue.cc:237] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.640216 20541 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.640359 20649 raft_consensus.cc:1273] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:34.644965 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.575556 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:34.650275 20497 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 13:58:34.618964 22096 raft_consensus.cc:3053] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:34.618975 20642 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
W20250831 13:58:34.630722 21645 consensus_meta.cc:287] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.836s user 0.010s sys 0.001s
I20250831 13:58:34.449569 22149 consensus_queue.cc:260] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.664453 20541 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.527253 22066 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:34.665575 22200 raft_consensus.cc:695] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:34.665733 22184 consensus_queue.cc:260] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.669653 21985 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:34.631515 22190 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:34.652545 22067 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:34.692059 20541 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:34.695786 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 57114.8us
I20250831 13:58:34.720171 20617 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:34.631856 22201 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:34.720785 20541 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.662967 22096 raft_consensus.cc:738] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:34.679180 22075 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:34.665395 21731 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:34.631171 21596 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.726043 20617 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:34.696273 22203 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:34.638743 22151 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20250831 13:58:34.730515 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:34.721655 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 159470us
W20250831 13:58:34.725617 21932 consensus_meta.cc:287] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.554s user 0.011s sys 0.004s
I20250831 13:58:34.562937 22003 consensus_queue.cc:237] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.724341 21715 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:34.717753 21914 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:34.690466 21988 consensus_queue.cc:1046] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20250831 13:58:34.734597 21588 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:34.735551 22155 consensus_meta.cc:287] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.935s user 0.011s sys 0.000s
I20250831 13:58:34.747012 20541 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:34.765242 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:34.740444 22208 consensus_queue.cc:1035] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20250831 13:58:34.725044 22105 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:34.734987 22184 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:34.679935 21645 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.675834 22148 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20250831 13:58:34.690140 21580 consensus_queue.cc:1046] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20250831 13:58:34.760916 20501 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20250831 13:58:34.663810 21565 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.766518 20541 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:34.736406 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 45545.3us
I20250831 13:58:34.752892 22193 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:34.759946 22075 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:34.777885 20499 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:34.760181 21896 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.791438 21708 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:34.760911 22069 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:34.696605 22162 consensus_queue.cc:237] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.754902 22067 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
I20250831 13:58:34.782553 22075 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:34.791921 22159 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:34.769483 22200 consensus_queue.cc:237] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:34.789140 22100 consensus_meta.cc:287] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.617s user 0.008s sys 0.000s
I20250831 13:58:34.771667 21731 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:34.775390 21715 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 13:58:34.782668 22198 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.252s user 0.001s sys 0.000s
I20250831 13:58:34.779001 21473 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.676309 22189 consensus_queue.cc:260] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.803354 21708 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:34.804098 21645 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:34.833626 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 107643us
I20250831 13:58:34.804911 22120 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:34.806156 22194 log.cc:927] Time spent T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
W20250831 13:58:34.806218 21609 consensus_meta.cc:287] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.677s user 0.010s sys 0.000s
I20250831 13:58:34.772121 22096 consensus_queue.cc:260] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.737762 22201 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.780203 22105 raft_consensus.cc:695] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:34.795615 21988 raft_consensus.cc:3053] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:34.771487 22155 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:34.801749 22075 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:34.781425 22066 consensus_queue.cc:1046] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
W20250831 13:58:34.805495 22207 log.cc:927] Time spent T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250831 13:58:34.800040 21896 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:34.813171 22195 log.cc:927] Time spent T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.285s user 0.000s sys 0.001s
I20250831 13:58:34.753700 22203 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.858487 22192 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:34.860126 22069 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 13:58:34.860478 22203 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:34.812996 22158 consensus_meta.cc:287] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.814s user 0.000s sys 0.010s
I20250831 13:58:34.861292 22158 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.863797 20644 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
W20250831 13:58:34.836463 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 157667us
I20250831 13:58:34.864370 20644 raft_consensus.cc:604] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:34.836556 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:34.821216 21580 raft_consensus.cc:3053] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:34.843739 20617 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:34.820107 21565 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.813139 22100 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:34.878110 20617 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:34.878504 20569 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:34.835391 21731 raft_consensus.cc:695] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:34.853879 22066 raft_consensus.cc:3053] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:34.849751 21988 raft_consensus.cc:738] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:34.854921 22201 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.849375 21609 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:34.873919 21209 raft_consensus.cc:2391] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
W20250831 13:58:34.877568 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:34.850353 22189 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:34.896809 20543 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.660s (SENT) [suppressed 29 similar messages]
W20250831 13:58:35.005507 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 441455us
I20250831 13:58:34.881654 20502 raft_consensus.cc:2366] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:34.854825 22120 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:34.882699 20617 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:34.899988 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:34.922188 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:34.875783 22127 raft_consensus.cc:3053] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:34.874291 21580 raft_consensus.cc:738] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:35.050472 20467 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:35.051149 20644 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:35.051796 20644 raft_consensus.cc:604] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:34.882496 20493 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:34.883853 22066 raft_consensus.cc:738] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:34.891220 21645 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:34.891108 21715 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:34.899446 21914 raft_consensus.cc:695] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:34.876123 21896 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:35.064426 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 281315us
I20250831 13:58:35.050547 22096 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:35.018646 22127 raft_consensus.cc:738] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:35.050563 21596 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.070518 20541 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:35.095223 20466 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:35.065021 22225 log.cc:927] Time spent T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20250831 13:58:35.061444 21645 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:35.061869 21565 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
W20250831 13:58:35.115916 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 46651.3us
I20250831 13:58:35.117341 20466 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:35.061628 21715 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:35.127347 20573 raft_consensus.cc:1273] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:35.130473 20569 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:35.064769 22158 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.136844 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:35.144671 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.137181 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.145562 20644 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:35.146126 20644 raft_consensus.cc:604] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:35.146507 20644 raft_consensus.cc:612] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:58:35.142596 21715 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:35.061558 22095 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20250831 13:58:35.148591 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:35.148669 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 266091us
W20250831 13:58:35.146792 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 249991us
I20250831 13:58:35.141207 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.151000 20576 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:35.153095 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:35.153561 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.156188 20617 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.120627 21645 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:35.109604 21988 consensus_queue.cc:260] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.139997 21596 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:35.165966 20649 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:35.160202 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:35.162988 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 45749.2us
I20250831 13:58:35.130820 22217 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:35.168641 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:35.164885 20576 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 13:58:35.145750 22219 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:35.061786 22105 consensus_queue.cc:237] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:35.163026 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:35.153497 22214 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 13:58:35.067951 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 696732us
I20250831 13:58:35.158037 22100 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.191810 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52966 (request call id 411) took 1477 ms. Trace:
I20250831 13:58:35.197005 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.197283 20494 rpcz_store.cc:276] 0831 13:58:33.714111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.714252 (+ 141us) service_pool.cc:225] Handling call
0831 13:58:35.180158 (+1465906us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000113d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8facd0 00007fb9bc97eeea 00007fb9c18d760e 00007fb9baf74cc7 00007fb9baf74c52 00007fb9baf74b81 00007fb9baf74afc 00007fb9baf73db2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:35.191757 (+ 11599us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2682501888}
I20250831 13:58:35.192709 20498 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:35.185905 20466 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:35.193172 20617 leader_election.cc:400] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6.
I20250831 13:58:35.199218 20617 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.118947 21580 consensus_queue.cc:260] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.200350 22214 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250831 13:58:35.168884 22217 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:35.200644 21609 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:35.201026 22217 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:35.202258 21176 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.200742 21580 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:35.205260 20617 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.208634 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:35.168430 21565 raft_consensus.cc:695] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:35.216264 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.178229 22186 raft_consensus.cc:3053] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
W20250831 13:58:35.219601 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:35.173848 22230 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:35.140622 21731 consensus_queue.cc:237] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.173861 22203 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:35.206233 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:35.223173 20569 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:35.179574 21896 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.158118 22127 consensus_queue.cc:260] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.168942 22221 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:35.225639 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.240082 20498 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 6
I20250831 13:58:35.212750 21988 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:35.229921 20466 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:35.204468 21715 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.234300 20494 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "90a0a56228d04660b3cb351485be6593"
dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
mode: GRACEFUL
new_leader_uuid: "ab7ade51fa134a1b814fab748f01d43d"
from {username='slave'} at 127.0.0.1:35300
I20250831 13:58:35.242998 20494 raft_consensus.cc:604] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Received request to transfer leadership to TS ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:35.243317 22227 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:35.146922 22066 consensus_queue.cc:260] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.244174 22229 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:35.244612 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:35.244773 22069 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:35.245289 22095 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:35.176549 22220 consensus_queue.cc:1046] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250831 13:58:35.245348 22069 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:35.168713 21914 consensus_queue.cc:237] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.239044 20569 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250831 13:58:35.238454 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.243239 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.249795 20466 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:35.238541 20617 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:35.235998 20571 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:35.175181 22159 consensus_queue.cc:1046] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20250831 13:58:35.225003 22186 raft_consensus.cc:738] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:35.248265 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:35.255808 21209 raft_consensus.cc:2391] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
I20250831 13:58:35.249941 20576 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:58:35.244123 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:35.253046 20617 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.259107 20466 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:35.259251 20617 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.262337 20466 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:35.229853 22120 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20250831 13:58:35.195170 22219 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.269475 22231 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:35.270332 22215 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:35.270797 22209 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:35.254516 21596 consensus_queue.cc:1046] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20250831 13:58:35.271482 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:35.271986 22223 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:35.262549 22229 consensus_queue.cc:1046] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20250831 13:58:35.254308 21896 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:35.271248 22209 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.234086 22203 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.272089 21208 raft_consensus.cc:3058] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:58:35.311710 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:35.264621 22125 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:35.267767 22181 log.cc:927] Time spent T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 1.036s user 0.014s sys 0.037s
I20250831 13:58:35.319185 21992 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:35.319994 22158 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:35.323531 20466 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:35.325443 20569 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:35.235266 22162 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20250831 13:58:35.256261 22230 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:35.267459 21932 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:35.336966 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:35.338079 20573 raft_consensus.cc:1273] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:35.343475 20498 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
W20250831 13:58:35.313669 22190 consensus_meta.cc:287] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.589s user 0.000s sys 0.010s
I20250831 13:58:35.274845 22227 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:35.268003 21715 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:35.274325 22205 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:35.361879 22212 log.cc:927] Time spent T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.370s user 0.000s sys 0.001s
I20250831 13:58:35.362756 22152 consensus_queue.cc:1046] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250831 13:58:35.363755 20502 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 674) took 1606 ms (client timeout 1660 ms). Trace:
I20250831 13:58:35.356781 22144 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20250831 13:58:35.364358 22222 log.cc:927] Time spent T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
I20250831 13:58:35.365361 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:35.266283 22155 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:35.360944 20541 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.379043 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.380769 21209 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:35.384452 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.386453 20494 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:35.387661 20502 rpcz_store.cc:269] 0831 13:58:33.757520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.757637 (+ 117us) service_pool.cc:225] Handling call
0831 13:58:35.363523 (+1605886us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000113d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:35.363672 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2536853888}
I20250831 13:58:35.413090 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.339278 22125 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:35.273777 21565 consensus_queue.cc:237] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.383598 21609 raft_consensus.cc:3053] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:35.383965 22190 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.471833 22190 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:35.473232 20649 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:58:35.392452 21983 consensus_meta.cc:287] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.961s user 0.008s sys 0.003s
I20250831 13:58:35.416496 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 692) took 1023 ms. Trace:
I20250831 13:58:35.416289 20617 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.476522 20651 raft_consensus.cc:2408] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:35.418116 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:35.477105 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 60927.5us
I20250831 13:58:35.477509 22203 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.478137 20617 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.479398 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:35.480511 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:35.419126 22125 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:35.481036 20646 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:35.481457 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:35.419075 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.483022 22190 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:35.482448 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:35.483713 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.485100 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:35.422503 22234 log.cc:927] Time spent T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
W20250831 13:58:35.424793 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 63980us
I20250831 13:58:35.438895 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 682) took 1570 ms. Trace:
W20250831 13:58:35.442011 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:35.486792 20643 rpcz_store.cc:276] 0831 13:58:33.868326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.868817 (+ 491us) service_pool.cc:225] Handling call
0831 13:58:33.894432 (+ 25615us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:34.307585 (+413153us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:35.438613 (+1131028us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b5400373a54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:35.438826 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":808185216}
I20250831 13:58:35.421456 22238 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:35.444296 22226 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.244s user 0.000s sys 0.001s
I20250831 13:58:35.444974 22123 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:35.445717 22219 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.446717 22127 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:35.447365 22069 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:35.454622 22184 consensus_meta.cc:287] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.677s user 0.000s sys 0.009s
W20250831 13:58:35.458346 22178 consensus_meta.cc:287] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.876s user 0.010s sys 0.000s
I20250831 13:58:35.459161 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 705) took 1022 ms. Trace:
I20250831 13:58:35.462015 20645 raft_consensus.cc:1273] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:35.463646 22209 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:35.403898 22213 log.cc:927] Time spent T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.373s user 0.000s sys 0.001s
I20250831 13:58:35.468852 20466 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:35.476178 20500 rpcz_store.cc:276] 0831 13:58:34.392839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.392979 (+ 140us) service_pool.cc:225] Handling call
0831 13:58:34.425145 (+ 32166us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:34.449174 (+ 24029us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:34.449208 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:35.154526 (+705318us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b2c00035a28. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bb3199a1 00007fb9bb328fd2 00007fb9bc9b2021 00007fb9bc9763fa 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32
0831 13:58:35.154857 (+ 331us) log.cc:844] Serialized 26 byte log entry
0831 13:58:35.253530 (+ 98673us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:35.253562 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:35.253618 (+ 56us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:35.410379 (+156761us) raft_consensus.cc:1683] finished
0831 13:58:35.410400 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:35.416411 (+ 6011us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16482,"raft.queue_time_us":62112,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":43795,"spinlock_wait_cycles":98066688,"thread_start_us":73257,"threads_started":1,"wal-append.queue_time_us":240326}
I20250831 13:58:35.480976 21983 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:35.399009 20576 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:35.424013 22236 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:35.489588 20541 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:35.493031 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:35.497608 20617 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.470502 22193 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:35.534006 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:35.485864 22197 log.cc:927] Time spent T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.678s user 0.010s sys 0.000s
I20250831 13:58:35.548851 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:35.549856 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 693) took 1156 ms. Trace:
I20250831 13:58:35.565578 20496 rpcz_store.cc:276] 0831 13:58:34.393279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.393404 (+ 125us) service_pool.cc:225] Handling call
0831 13:58:34.427711 (+ 34307us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:34.518548 (+ 90837us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:34.518596 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:34.563823 (+ 45227us) log.cc:844] Serialized 26 byte log entry
0831 13:58:34.713388 (+149565us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:34.713444 (+ 56us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:34.713532 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:35.549604 (+836072us) raft_consensus.cc:1683] finished
0831 13:58:35.549630 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:35.549788 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":91817,"raft.queue_time_us":91130,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":7113,"spinlock_wait_cycles":50048,"thread_start_us":71974,"threads_started":1,"wal-append.queue_time_us":159726}
I20250831 13:58:35.466145 21609 raft_consensus.cc:738] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:35.550783 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 703) took 1114 ms. Trace:
I20250831 13:58:35.552067 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:35.491128 20642 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:35.559104 20648 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:35.560161 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:35.454295 22217 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:35.563454 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 94707.1us
I20250831 13:58:35.566988 20497 rpcz_store.cc:276] 0831 13:58:34.436454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.660283 (+223829us) service_pool.cc:225] Handling call
0831 13:58:34.660614 (+ 331us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:34.800393 (+139779us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:34.800425 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:34.800852 (+ 427us) log.cc:844] Serialized 26 byte log entry
0831 13:58:35.020814 (+219962us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:35.020869 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:35.181510 (+160641us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54001289d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:35.181567 (+ 57us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:35.523736 (+342169us) raft_consensus.cc:1683] finished
0831 13:58:35.523757 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:35.550700 (+ 26943us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2517,"raft.queue_time_us":10644,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5994,"spinlock_wait_cycles":320712576,"thread_start_us":219636,"threads_started":1,"wal-append.queue_time_us":386007}
I20250831 13:58:35.568599 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 696) took 1174 ms. Trace:
I20250831 13:58:35.523125 21473 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:35.571450 20568 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:35.572417 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.581619 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.573689 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.576097 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.583006 20573 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:35.577106 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:35.378971 22186 consensus_queue.cc:260] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.577020 20501 rpcz_store.cc:276] 0831 13:58:34.394471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.394590 (+ 119us) service_pool.cc:225] Handling call
0831 13:58:34.394973 (+ 383us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:34.777753 (+382780us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:35.568301 (+790548us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b5400099cd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:35.568529 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":840932736}
I20250831 13:58:35.586759 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:35.587257 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.587472 20497 raft_consensus.cc:2391] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:35.579160 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 81659.9us
I20250831 13:58:35.588474 20497 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:35.588585 20617 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.589017 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 710) took 1068 ms. Trace:
I20250831 13:58:35.589509 20498 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:35.589666 20617 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.589344 20497 rpcz_store.cc:276] 0831 13:58:34.520692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.587954 (+1067262us) service_pool.cc:225] Handling call
0831 13:58:35.588186 (+ 232us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:35.588796 (+ 610us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:35.588972 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:35.591349 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.591859 20500 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:35.579636 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 90151.2us
I20250831 13:58:35.585057 20570 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:35.591216 20497 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:35.592263 20498 raft_consensus.cc:1273] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20250831 13:58:35.579197 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 348542us
I20250831 13:58:35.596315 20498 pending_rounds.cc:85] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 1
I20250831 13:58:35.596403 20617 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.596818 20498 pending_rounds.cc:107] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 13:58:35.597223 20498 raft_consensus.cc:2887] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 13:58:35.597504 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:35.597950 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:35.523979 22190 consensus_queue.cc:1046] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20250831 13:58:35.599270 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:35.599987 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:35.557025 21992 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:35.605834 20497 raft_consensus.cc:1238] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:35.607697 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:35.608368 20643 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:35.610560 22246 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:35.572924 22209 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:35.612119 20576 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:35.613059 20498 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
W20250831 13:58:35.618659 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:35.615868 20617 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:35.695837 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 80046us
I20250831 13:58:35.697679 20498 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:35.699060 20572 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:35.698652 20617 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.585970 21983 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:35.700421 20498 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 13:58:35.703297 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:35.708020 20617 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.711580 20500 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:35.556905 21174 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:35.523418 22238 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:35.575841 22066 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:35.577883 22193 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.599056 21565 consensus_queue.cc:1035] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.100s
I20250831 13:58:35.694434 22205 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:35.730572 22075 consensus_meta.cc:287] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.875s user 0.009s sys 0.000s
I20250831 13:58:35.715360 22214 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250831 13:58:35.732282 22096 consensus_meta.cc:287] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.662s user 0.007s sys 0.000s
I20250831 13:58:35.602159 22178 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:35.609509 22184 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:35.614262 21988 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:35.599216 21473 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.614096 21985 consensus_queue.cc:1046] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.127s
I20250831 13:58:35.723518 22206 raft_consensus.cc:3053] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:35.723426 22186 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:35.715642 21715 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:35.733494 21580 consensus_meta.cc:287] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.520s user 0.008s sys 0.000s
I20250831 13:58:35.727654 22190 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:35.731113 21941 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:35.613744 22155 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.128s
I20250831 13:58:35.728614 22209 raft_consensus.cc:695] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:35.742270 22078 consensus_meta.cc:287] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 1.379s user 0.012s sys 0.000s
I20250831 13:58:35.735951 22069 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:35.740530 22075 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.713375 21983 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.765024 22075 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:35.756268 22206 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:35.753957 22178 consensus_queue.cc:1035] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20250831 13:58:35.765971 22206 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.616191 21992 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.755185 22205 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:35.754405 21767 raft_consensus.cc:3053] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
W20250831 13:58:35.902725 21645 consensus_meta.cc:287] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.732s user 0.004s sys 0.005s
I20250831 13:58:35.903190 21731 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:35.903801 21645 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:35.905174 21992 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.777186 22125 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.824579 20398 catalog_manager.cc:5582] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:35.601023 20466 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:35.831117 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:35.831291 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:35.829835 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:35.714401 20498 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:36.290236 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:35.846995 20502 raft_consensus.cc:2408] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:35.551314 20493 rpcz_store.cc:276] 0831 13:58:34.437015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.735296 (+298281us) service_pool.cc:225] Handling call
0831 13:58:34.761813 (+ 26517us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:35.059604 (+297791us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:35.458893 (+399289us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54000122d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:35.459105 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":231617792}
I20250831 13:58:35.667404 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:35.841800 20499 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 697) took 1446 ms (client timeout 1581 ms). Trace:
I20250831 13:58:35.831149 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 695) took 1436 ms. Trace:
I20250831 13:58:36.509245 20495 rpcz_store.cc:276] 0831 13:58:34.394136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.394575 (+ 439us) service_pool.cc:225] Handling call
0831 13:58:34.394973 (+ 398us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:34.414967 (+ 19994us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:34.414995 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:34.429349 (+ 14354us) log.cc:844] Serialized 26 byte log entry
0831 13:58:34.744513 (+315164us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:34.900598 (+156085us) log.cc:844] Serialized 14 byte log entry
0831 13:58:34.900904 (+ 306us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:34.901003 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:35.830221 (+929218us) raft_consensus.cc:1683] finished
0831 13:58:35.830312 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:35.830850 (+ 538us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55472,"prepare.queue_time_us":450444,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":33561,"raft.run_cpu_time_us":1576,"raft.run_wall_time_us":31660,"spinlock_wait_cycles":86400,"thread_start_us":321408,"threads_started":2,"wal-append.queue_time_us":279551}
I20250831 13:58:35.864660 20501 raft_consensus.cc:1238] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:35.872099 20543 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:36.518222 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 646558us
I20250831 13:58:35.849630 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:35.839607 20494 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20250831 13:58:35.857196 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 140747us
I20250831 13:58:35.739977 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:36.536311 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 981) took 1083 ms. Trace:
I20250831 13:58:36.536759 20643 rpcz_store.cc:276] 0831 13:58:35.452395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.714488 (+262093us) service_pool.cc:225] Handling call
0831 13:58:36.536112 (+821624us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.538056 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:35.852272 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.531498 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:36.539009 22178 consensus_queue.cc:1046] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:36.539549 20644 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:36.539978 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37606 (request call id 35) took 1172 ms. Trace:
I20250831 13:58:36.540189 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 712) took 1385 ms. Trace:
I20250831 13:58:36.540403 20494 rpcz_store.cc:276] 0831 13:58:35.368024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.838443 (+470419us) service_pool.cc:225] Handling call
0831 13:58:35.838944 (+ 501us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:36.539715 (+700771us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.539928 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.540592 20644 rpcz_store.cc:276] 0831 13:58:35.154740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.851572 (+696832us) service_pool.cc:225] Handling call
0831 13:58:36.540126 (+688554us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.541390 22178 raft_consensus.cc:3053] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:36.541766 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:35.864754 20647 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20250831 13:58:36.544286 20647 pending_rounds.cc:85] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Aborting all ops after (but not including) 1
I20250831 13:58:36.544600 20647 pending_rounds.cc:107] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20250831 13:58:36.544847 20647 raft_consensus.cc:2887] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:36.545883 20617 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:35.749835 22256 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:35.739450 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.553406 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 990) took 1085 ms. Trace:
I20250831 13:58:36.553737 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:36.553779 20642 rpcz_store.cc:276] 0831 13:58:35.467855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.713141 (+245286us) service_pool.cc:225] Handling call
0831 13:58:36.553225 (+840084us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.554725 20497 raft_consensus.cc:2391] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:35.849542 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 973) took 1439 ms. Trace:
I20250831 13:58:36.555958 20648 rpcz_store.cc:276] 0831 13:58:34.409779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.453497 (+ 43718us) service_pool.cc:225] Handling call
0831 13:58:34.519523 (+ 66026us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:35.569550 (+1050027us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:35.848899 (+279349us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400010254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:35.849229 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2390039424}
I20250831 13:58:36.557440 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.557574 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:36.558136 20648 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:35.855717 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 969) took 1483 ms. Trace:
I20250831 13:58:35.861450 20650 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:36.561892 20497 raft_consensus.cc:1238] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
W20250831 13:58:35.863278 20651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 949) took 1540 ms (client timeout 1698 ms). Trace:
W20250831 13:58:36.068212 20646 consensus_meta.cc:287] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.586s user 0.013s sys 0.009s
I20250831 13:58:36.564405 20646 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
W20250831 13:58:36.564260 20651 rpcz_store.cc:269] 0831 13:58:34.322865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.322978 (+ 113us) service_pool.cc:225] Handling call
0831 13:58:35.862720 (+1539742us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540028b754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:35.863032 (+ 312us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2545181568}
I20250831 13:58:36.565158 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 987) took 1111 ms. Trace:
I20250831 13:58:36.565537 20651 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
W20250831 13:58:36.145610 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:36.120775 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:36.164887 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:36.176137 20496 consensus_meta.cc:287] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.575s user 0.023s sys 0.003s
I20250831 13:58:36.187930 20649 raft_consensus.cc:2366] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7.
I20250831 13:58:36.276868 20397 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59924 (request call id 138) took 9841 ms. Trace:
I20250831 13:58:36.558599 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 719) took 1034 ms. Trace:
I20250831 13:58:36.276173 20398 catalog_manager.cc:5582] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: config changed from index 2 to 5, term changed from 3 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
W20250831 13:58:36.277733 20574 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.015s sys 0.008s
W20250831 13:58:36.568022 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:36.568440 20574 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.977s user 0.016s sys 0.008s
W20250831 13:58:36.292063 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 690990us
I20250831 13:58:35.841682 20500 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
W20250831 13:58:35.747952 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 224692us
I20250831 13:58:35.753429 21208 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:36.578083 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47732 (request call id 114) took 1308 ms. Trace:
I20250831 13:58:35.752314 21473 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.901623 21588 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:35.968794 21645 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:36.668184 21645 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:36.668781 21645 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.567545 20397 rpcz_store.cc:276] 0831 13:58:26.435232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:26.435555 (+ 323us) service_pool.cc:225] Handling call
0831 13:58:36.276634 (+9841079us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:33.519734 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:33.520286 (+ 552us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:33.520325 (+ 39us) write_op.cc:435] Acquired schema lock
0831 13:58:33.520341 (+ 16us) tablet.cc:660] Decoding operations
0831 13:58:33.534859 (+ 14518us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:33.534911 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:33.534938 (+ 27us) tablet.cc:683] Acquiring locks for 6 operations
0831 13:58:33.535398 (+ 460us) tablet.cc:699] Row locks acquired
0831 13:58:33.535414 (+ 16us) write_op.cc:260] PREPARE: finished
0831 13:58:33.535570 (+ 156us) write_op.cc:270] Start()
0831 13:58:33.535672 (+ 102us) write_op.cc:276] Timestamp: P: 1756648713535514 usec, L: 0
0831 13:58:33.535690 (+ 18us) op_driver.cc:348] REPLICATION: starting
0831 13:58:33.635507 (+ 99817us) log.cc:844] Serialized 2499 byte log entry
0831 13:58:34.228673 (+593166us) op_driver.cc:464] REPLICATION: finished
0831 13:58:34.607643 (+378970us) write_op.cc:301] APPLY: starting
0831 13:58:34.620307 (+ 12664us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:34.663022 (+ 42715us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:34.674848 (+ 11826us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:34.728875 (+ 54027us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:34.758512 (+ 29637us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0831 13:58:34.768126 (+ 9614us) write_op.cc:312] APPLY: finished
0831 13:58:34.798472 (+ 30346us) log.cc:844] Serialized 52 byte log entry
0831 13:58:34.810969 (+ 12497us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:34.834982 (+ 24013us) write_op.cc:454] Released schema lock
0831 13:58:34.835656 (+ 674us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:34.856661 (+ 21005us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:34.856705 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3332010,"reported_tablets":28,"spinlock_wait_cycles":934272,"child_traces":[["op",{"apply.queue_time_us":378754,"apply.run_cpu_time_us":6384,"apply.run_wall_time_us":241480,"num_ops":6,"prepare.queue_time_us":179579,"prepare.run_cpu_time_us":39014,"prepare.run_wall_time_us":174070,"raft.queue_time_us":216661,"raft.run_cpu_time_us":20336,"raft.run_wall_time_us":200279,"replication_time_us":692853,"thread_start_us":475255,"threads_started":4,"wal-append.queue_time_us":104507,"wal-append.run_cpu_time_us":21465,"wal-append.run_wall_time_us":2220214}]]}
I20250831 13:58:35.902725 21708 consensus_queue.cc:1046] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
I20250831 13:58:35.761407 21715 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.684235 21715 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.684775 21715 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:36.685273 21715 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.706790 21609 consensus_queue.cc:260] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.903230 21767 raft_consensus.cc:738] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:36.714179 21767 consensus_queue.cc:260] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.715984 21767 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:35.767293 22078 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.718209 20466 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:36.192581 22080 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:36.721387 22078 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.951591 22125 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.051031 22149 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:35.787484 22159 raft_consensus.cc:3053] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:36.727973 22159 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:36.729138 22080 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.728974 22159 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.730088 22159 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:36.121351 22182 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:36.731308 22182 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.734160 22182 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:36.734354 22295 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.153342 22096 raft_consensus.cc:3053] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:36.735162 22125 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:36.735399 22096 raft_consensus.cc:738] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:36.735690 22125 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:35.761878 22190 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.737787 20466 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:35.785876 22193 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:35.789019 22202 raft_consensus.cc:3053] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:36.740588 22202 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:36.741495 22202 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.780157 22217 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:35.782492 22223 consensus_meta.cc:287] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.510s user 0.005s sys 0.003s
I20250831 13:58:36.745755 22223 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:36.745975 22221 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:35.775617 22196 raft_consensus.cc:991] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: : Instructing follower ab7ade51fa134a1b814fab748f01d43d to start an election
I20250831 13:58:36.748821 22196 raft_consensus.cc:1079] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Signalling peer ab7ade51fa134a1b814fab748f01d43d to start an election
I20250831 13:58:36.307711 20575 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:36.290534 22210 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:35.766978 22206 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:35.787423 22209 consensus_queue.cc:237] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:35.795003 22231 raft_consensus.cc:695] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:36.309696 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:35.794826 22233 raft_consensus.cc:3053] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:36.761638 22233 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:36.761358 22231 consensus_queue.cc:237] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.762632 22233 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.763722 22233 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:35.779528 22238 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:36.388543 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52966 (request call id 412) took 1175 ms. Trace:
I20250831 13:58:36.766474 20502 rpcz_store.cc:276] 0831 13:58:35.213105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.299188 (+1086083us) service_pool.cc:225] Handling call
0831 13:58:36.388454 (+ 89266us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.766883 20617 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:36.391626 20570 raft_consensus.cc:1273] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250831 13:58:36.395982 20387 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 96225.9us
I20250831 13:58:36.397120 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.404642 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:36.406565 21932 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.738617 22190 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.770632 22190 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:36.419109 22127 consensus_queue.cc:1046] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:36.428438 22100 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.432808 22067 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.478454 22216 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:36.482124 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 51965.9us
I20250831 13:58:36.497423 22157 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.779009 22223 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.779742 22223 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.780668 22295 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.783165 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:36.784344 21205 raft_consensus.cc:2391] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:36.502341 20499 rpcz_store.cc:269] 0831 13:58:34.394846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.451620 (+ 56774us) service_pool.cc:225] Handling call
0831 13:58:35.841313 (+1389693us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b540009a6d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:35.841550 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":331356032}
I20250831 13:58:36.513171 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1114) took 1193 ms. Trace:
I20250831 13:58:36.519773 21914 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:36.542559 22220 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:36.541803 22178 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:36.787551 22220 raft_consensus.cc:695] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:36.788302 22178 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.788659 22220 consensus_queue.cc:237] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.789434 22178 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:36.549984 22256 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:36.542559 20644 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:36.559296 20645 rpcz_store.cc:276] 0831 13:58:34.372258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.425460 (+ 53202us) service_pool.cc:225] Handling call
0831 13:58:34.425729 (+ 269us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:35.551506 (+1125777us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:35.855004 (+303498us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400373054. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:35.855466 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2382663936}
I20250831 13:58:36.565569 20646 rpcz_store.cc:276] 0831 13:58:35.453646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.480298 (+ 26652us) service_pool.cc:225] Handling call
0831 13:58:36.565079 (+1084781us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.806329 20569 raft_consensus.cc:2366] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7.
I20250831 13:58:36.154510 20577 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:36.556171 20495 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:36.806977 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:36.807653 20645 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:36.807960 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.808575 22223 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.569666 22199 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:36.571126 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:36.810411 20497 raft_consensus.cc:2366] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7.
I20250831 13:58:36.572696 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46464 (request call id 38) took 1107 ms. Trace:
I20250831 13:58:36.567956 20648 rpcz_store.cc:276] 0831 13:58:35.523736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.557119 (+1033383us) service_pool.cc:225] Handling call
0831 13:58:36.558532 (+ 1413us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.572990 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:36.574139 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 976) took 2049 ms. Trace:
I20250831 13:58:36.575806 22270 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 13:58:36.551508 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.592046 22066 consensus_queue.cc:1046] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:36.814455 20494 raft_consensus.cc:2366] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:36.814638 22066 raft_consensus.cc:3053] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:36.601629 22278 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:36.604548 20496 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:36.608383 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:36.609329 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1120) took 1287 ms. Trace:
I20250831 13:58:36.811192 20647 rpcz_store.cc:276] 0831 13:58:34.524525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:34.607505 (+ 82980us) service_pool.cc:225] Handling call
0831 13:58:35.863831 (+1256326us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:36.545655 (+681824us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.573836 (+ 28181us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540028b750. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:36.574061 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2888438528}
I20250831 13:58:36.578449 21208 rpcz_store.cc:276] 0831 13:58:35.269751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.271241 (+ 1490us) service_pool.cc:225] Handling call
0831 13:58:36.577885 (+1306644us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.639735 22289 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:36.650673 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 88281.2us
I20250831 13:58:36.655373 22286 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.655871 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1057) took 1414 ms. Trace:
I20250831 13:58:36.655954 22095 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.656159 21580 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.815622 20500 rpcz_store.cc:276] 0831 13:58:35.321472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.712490 (+391018us) service_pool.cc:225] Handling call
0831 13:58:35.841059 (+128569us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:36.574929 (+733870us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.607363 (+ 32434us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400186fd0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:36.609252 (+ 1889us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":294436096}
I20250831 13:58:36.656411 20541 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:36.648450 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:36.817517 20577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1114) took 1680 ms (client timeout 2141 ms). Trace:
I20250831 13:58:36.663578 22288 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:36.826305 21914 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.665120 22229 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:36.665467 20650 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:36.666280 21588 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.680765 22292 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:36.827610 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 245853us
I20250831 13:58:36.828346 22292 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.830401 22292 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.682179 21645 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.689095 21715 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.699419 22291 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:36.715191 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:36.834874 20645 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:36.145540 21659 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.839823 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:36.181881 21983 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.827086 21580 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:36.844946 21580 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:36.826153 22289 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.846557 22289 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.832787 20642 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:36.847746 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 717) took 1353 ms. Trace:
I20250831 13:58:36.848057 20642 rpcz_store.cc:276] 0831 13:58:35.494608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.554651 (+1060043us) service_pool.cc:225] Handling call
0831 13:58:36.847621 (+292970us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b5400373054. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:36.847702 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14103808}
I20250831 13:58:36.848959 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:36.849206 20500 consensus_queue.cc:237] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.850509 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 707) took 1697 ms. Trace:
I20250831 13:58:36.717124 21565 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:36.826965 22288 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.828354 21914 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:35.615609 22003 consensus_queue.cc:1162] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "212b66976f874aa682cae0fb77928786" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 13:58:36.717113 22294 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.719139 20573 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:36.726745 21609 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:36.730584 22293 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.292129 22152 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:35.956389 22188 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.736141 22125 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:36.738802 22146 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 13:58:36.739030 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:36.858628 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:36.746551 22221 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:36.753410 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:36.754175 22078 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.755534 21209 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "90a0a56228d04660b3cb351485be6593"
dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
from {username='slave'} at 127.0.0.1:47732
I20250831 13:58:36.875069 21209 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250831 13:58:36.875528 21209 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:36.756479 21767 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.751641 22210 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.754345 22202 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:36.876848 22186 consensus_queue.cc:1035] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:36.761178 20568 raft_consensus.cc:2366] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:36.770167 21932 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.880541 22210 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:36.880633 21932 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:36.416676 21992 consensus_queue.cc:1046] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:36.881691 21992 raft_consensus.cc:3053] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:36.881517 21767 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.882061 21992 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:36.771391 22127 consensus_queue.cc:1230] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer 212b66976f874aa682cae0fb77928786 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 13:58:36.775197 22100 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.790690 22216 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.883466 21767 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.884328 22100 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.884440 22216 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.785791 20501 rpcz_store.cc:276] 0831 13:58:35.319439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.606806 (+287367us) service_pool.cc:225] Handling call
0831 13:58:35.607043 (+ 237us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:35.864280 (+257237us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54001c1720. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc905550 00007fb9bc978170 00007fb9bc9791e1 00007fb9bc975d4f 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c
0831 13:58:36.512843 (+648563us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.513115 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":541023104}
I20250831 13:58:36.777951 22157 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.859704 21588 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:36.885200 20572 raft_consensus.cc:1273] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:36.886134 20575 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:36.886591 22157 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.887029 20541 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:36.887215 22157 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:36.887660 22157 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:36.887425 21992 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.887967 22221 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.888110 22157 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:36.888689 21992 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:36.888533 22078 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.791750 22067 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.890872 21588 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:36.867679 22188 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.891291 21588 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:36.891682 21588 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:36.892102 22067 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.806360 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 979) took 1354 ms. Trace:
I20250831 13:58:36.892637 22188 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.892796 20644 rpcz_store.cc:276] 0831 13:58:35.451928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.541434 (+1089506us) service_pool.cc:225] Handling call
0831 13:58:36.806271 (+264837us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.893766 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.809882 22199 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.862123 22293 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.896210 22199 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.813964 20648 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:36.896780 22293 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.897007 20616 leader_election.cc:400] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:36.898502 20616 leader_election.cc:403] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20250831 13:58:36.810760 20649 rpcz_store.cc:276] 0831 13:58:35.465565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.482213 (+ 16648us) service_pool.cc:225] Handling call
0831 13:58:36.572461 (+1090248us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54002ea754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:36.572643 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1617506048}
I20250831 13:58:36.898665 20468 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:36.899494 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:36.900590 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:36.900743 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.901407 20649 raft_consensus.cc:2391] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:36.853307 22288 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:36.662887 22281 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.821233 22295 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.809664 20646 raft_consensus.cc:1238] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:36.902690 22146 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:36.903157 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 704) took 1751 ms. Trace:
I20250831 13:58:36.903596 20646 rpcz_store.cc:276] 0831 13:58:35.152204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.808996 (+1656792us) service_pool.cc:225] Handling call
0831 13:58:36.809371 (+ 375us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:36.902908 (+ 93537us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.903106 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.904596 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.904930 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:36.905198 20575 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:36.905512 21205 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:36.673334 21708 raft_consensus.cc:3053] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:36.906152 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.906313 21708 raft_consensus.cc:738] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:36.894582 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.907306 21708 consensus_queue.cc:260] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.908501 21708 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:36.908684 22293 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:36.909423 21932 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.909946 21932 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.831171 21175 leader_election.cc:400] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7.
I20250831 13:58:36.910817 21175 leader_election.cc:403] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20250831 13:58:36.911370 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 80441.5us
I20250831 13:58:36.911254 22125 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.911845 21932 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.831055 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.912428 21932 raft_consensus.cc:2802] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:36.912586 20576 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:36.912930 21932 raft_consensus.cc:695] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:36.913502 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.913863 21932 consensus_queue.cc:237] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.832218 22278 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:36.832316 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 706) took 1679 ms. Trace:
W20250831 13:58:36.821748 20577 rpcz_store.cc:269] 0831 13:58:35.136716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.136860 (+ 144us) service_pool.cc:225] Handling call
0831 13:58:36.817309 (+1680449us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b540031d8d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:36.817452 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2302421888}
I20250831 13:58:36.826357 20543 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:36.826056 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 717) took 1600 ms. Trace:
I20250831 13:58:36.815028 22066 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:36.838971 22229 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.841609 20569 raft_consensus.cc:2391] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:36.837613 22190 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.822492 22095 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:36.837833 22291 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.827355 22286 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.851310 21565 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.729753 22080 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:36.944958 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 118699us
I20250831 13:58:36.853317 21914 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.856221 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:36.858168 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.850804 20647 rpcz_store.cc:276] 0831 13:58:35.152903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.821540 (+1668637us) service_pool.cc:225] Handling call
0831 13:58:36.826827 (+ 5287us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:36.849488 (+ 22661us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:36.849506 (+ 18us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:36.849546 (+ 40us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:36.849562 (+ 16us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.850345 (+ 783us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b540015aed4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:36.850371 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:36.850452 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51728128}
I20250831 13:58:36.853778 22294 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.744570 20466 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:36.992285 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 247864us
I20250831 13:58:36.993389 21992 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:36.993919 21992 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.861630 20398 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54880 (request call id 37) took 3142 ms. Trace:
I20250831 13:58:36.869695 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1123) took 1381 ms. Trace:
I20250831 13:58:36.995635 21992 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.753890 22182 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.996104 21992 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:36.996364 22182 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.996796 21992 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:36.997357 21914 consensus_queue.cc:1046] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:36.840153 21659 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.998517 22067 raft_consensus.cc:3053] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:36.998347 22202 consensus_queue.cc:1035] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:36.999064 22067 raft_consensus.cc:738] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.000018 21659 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:36.999927 22067 consensus_queue.cc:260] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.902093 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:36.816962 20574 rpcz_store.cc:276] 0831 13:58:35.239427 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.245418 (+ 5991us) service_pool.cc:225] Handling call
0831 13:58:35.245691 (+ 273us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:35.562866 (+317175us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:35.562896 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:35.563402 (+ 506us) log.cc:844] Serialized 26 byte log entry
0831 13:58:36.608888 (+1045486us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:36.636955 (+ 28067us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.649469 (+ 12514us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54002a8fd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:36.649540 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:36.649601 (+ 61us) raft_consensus.cc:1683] finished
0831 13:58:36.649620 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:36.653757 (+ 4137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19314,"raft.queue_time_us":33863,"raft.run_cpu_time_us":65,"raft.run_wall_time_us":65,"spinlock_wait_cycles":324136320,"thread_start_us":1132462,"threads_started":2,"wal-append.queue_time_us":419665}
I20250831 13:58:37.001345 22067 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:37.001472 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.995843 20573 rpcz_store.cc:276] 0831 13:58:35.488569 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.583714 (+ 95145us) service_pool.cc:225] Handling call
0831 13:58:35.584087 (+ 373us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:36.866326 (+1282239us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.869410 (+ 3084us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54003d40d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:36.869602 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2609509760}
I20250831 13:58:37.002602 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.002983 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.003758 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.902799 22281 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.004071 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.004586 22066 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.005133 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.005787 22281 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.005889 22066 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.006441 20574 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:36.902421 20651 raft_consensus.cc:604] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.007587 20651 raft_consensus.cc:612] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:58:37.008266 22229 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.994810 20398 rpcz_store.cc:276] 0831 13:58:33.719394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:33.719544 (+ 150us) service_pool.cc:225] Handling call
0831 13:58:36.861557 (+3142013us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:36.641007 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:36.641927 (+ 920us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:36.641993 (+ 66us) write_op.cc:435] Acquired schema lock
0831 13:58:36.642030 (+ 37us) tablet.cc:660] Decoding operations
0831 13:58:36.642870 (+ 840us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:36.643014 (+ 144us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:36.643053 (+ 39us) tablet.cc:683] Acquiring locks for 2 operations
0831 13:58:36.643552 (+ 499us) tablet.cc:699] Row locks acquired
0831 13:58:36.643580 (+ 28us) write_op.cc:260] PREPARE: finished
0831 13:58:36.643790 (+ 210us) write_op.cc:270] Start()
0831 13:58:36.643995 (+ 205us) write_op.cc:276] Timestamp: P: 1756648716643708 usec, L: 0
0831 13:58:36.644032 (+ 37us) op_driver.cc:348] REPLICATION: starting
0831 13:58:36.644926 (+ 894us) log.cc:844] Serialized 988 byte log entry
0831 13:58:36.746384 (+101458us) op_driver.cc:464] REPLICATION: finished
0831 13:58:36.838176 (+ 91792us) write_op.cc:301] APPLY: starting
0831 13:58:36.853375 (+ 15199us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:36.853510 (+ 135us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:36.853648 (+ 138us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:36.855751 (+ 2103us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:36.856050 (+ 299us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0831 13:58:36.856114 (+ 64us) write_op.cc:312] APPLY: finished
0831 13:58:36.856383 (+ 269us) log.cc:844] Serialized 28 byte log entry
0831 13:58:36.856555 (+ 172us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:36.856641 (+ 86us) write_op.cc:454] Released schema lock
0831 13:58:36.856691 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:36.857139 (+ 448us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:36.857172 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":366133,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":66709,"apply.run_cpu_time_us":2495,"apply.run_wall_time_us":35255,"num_ops":2,"prepare.queue_time_us":61177,"prepare.run_cpu_time_us":14919,"prepare.run_wall_time_us":30996,"raft.queue_time_us":29923,"raft.run_cpu_time_us":8504,"raft.run_wall_time_us":28404,"replication_time_us":102098,"thread_start_us":131051,"threads_started":4,"wal-append.queue_time_us":53014}]]}
I20250831 13:58:37.009534 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:41962 (request call id 8) took 1648 ms. Trace:
I20250831 13:58:37.009861 20651 rpcz_store.cc:276] 0831 13:58:35.361213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.565332 (+1204119us) service_pool.cc:225] Handling call
0831 13:58:37.009358 (+444026us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b5400069d54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc970813 00007fb9c18d61ba 00007fb9baf701c4 00007fb9baf70152 00007fb9baf70081 00007fb9baf6fffc 00007fb9baf6f2b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:37.009469 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":770672384}
I20250831 13:58:36.894405 20644 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:37.011669 20644 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 13:58:37.012226 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 709) took 1858 ms. Trace:
I20250831 13:58:37.012519 20644 rpcz_store.cc:276] 0831 13:58:35.154062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.011096 (+1857034us) service_pool.cc:225] Handling call
0831 13:58:37.011445 (+ 349us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:37.011978 (+ 533us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.012157 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.936975 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.897038 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 705) took 1744 ms. Trace:
I20250831 13:58:36.924142 20575 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:37.015177 20576 raft_consensus.cc:2408] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:37.015105 20648 rpcz_store.cc:276] 0831 13:58:35.152445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.813503 (+1661058us) service_pool.cc:225] Handling call
0831 13:58:36.813806 (+ 303us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:36.896789 (+ 82983us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:36.896967 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:37.015647 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.015977 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.016287 20648 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:37.016418 20575 raft_consensus.cc:2391] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:37.016801 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 713) took 1861 ms. Trace:
I20250831 13:58:37.017009 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.017136 20648 rpcz_store.cc:276] 0831 13:58:35.155010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.015809 (+1860799us) service_pool.cc:225] Handling call
0831 13:58:37.016074 (+ 265us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.016616 (+ 542us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.016743 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:36.933148 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:36.937906 20468 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:37.018271 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:37.018836 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 81003.8us
I20250831 13:58:37.018925 20651 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:36.936156 20496 rpcz_store.cc:276] 0831 13:58:35.225665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.566975 (+341310us) service_pool.cc:225] Handling call
0831 13:58:36.821490 (+1254515us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:36.915225 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.902189 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.936367 20650 rpcz_store.cc:276] 0831 13:58:35.152661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:35.152895 (+ 234us) service_pool.cc:225] Handling call
0831 13:58:36.832150 (+1679255us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b540006b154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:36.832259 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1623568256}
I20250831 13:58:36.915786 20643 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:37.020129 20651 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250831 13:58:37.020655 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46464 (request call id 35) took 1569 ms. Trace:
I20250831 13:58:36.916698 22078 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.947645 20467 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:37.020956 20651 rpcz_store.cc:276] 0831 13:58:35.451197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.019616 (+1568419us) service_pool.cc:225] Handling call
0831 13:58:37.019870 (+ 254us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.020456 (+ 586us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.020609 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:37.021593 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 74022.2us
I20250831 13:58:36.948559 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.022392 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 988) took 1568 ms. Trace:
I20250831 13:58:36.950645 21209 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:36.951913 20543 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:36.918975 22127 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:36.914852 22270 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7.
I20250831 13:58:36.939306 22278 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:36.958369 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:36.955592 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.919500 22152 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:36.962966 20645 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:36.904596 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.964789 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:36.967868 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:36.964881 20616 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:36.918644 20396 catalog_manager.cc:5582] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 5, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 5 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:36.953979 21175 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:36.976089 22095 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.979089 21565 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.980932 22291 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.963138 22190 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.027608 22221 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:37.055552 22221 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.055779 20649 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:37.055934 22221 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.057013 22190 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.983139 22286 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:36.965776 22238 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:37.058539 22238 raft_consensus.cc:695] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.031049 22202 consensus_queue.cc:1035] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20250831 13:58:37.058866 22270 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:36.954222 22178 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.059284 22270 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.059653 22270 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.059577 22178 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.985448 22294 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.948382 22216 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20250831 13:58:36.985549 21588 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.066102 20651 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:36.988464 22096 consensus_queue.cc:260] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.067078 22178 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.067446 22096 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:37.068128 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 117620us
I20250831 13:58:37.071008 22202 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.040894 22078 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:37.071499 22202 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.073463 22202 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.073849 22127 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.006760 20466 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.014586 22157 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:37.075143 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 80653.8us
I20250831 13:58:37.075753 22127 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.075812 20466 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:37.076475 22127 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:37.006132 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.076951 22127 raft_consensus.cc:695] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:37.025388 20467 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:37.078709 22202 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 13:58:37.079347 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 54052.1us
I20250831 13:58:37.080278 22202 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:37.080964 20467 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:37.026098 20572 raft_consensus.cc:2391] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 2: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:37.082482 22196 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:37.082530 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 980) took 1630 ms. Trace:
I20250831 13:58:37.083001 22196 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.083446 22196 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
W20250831 13:58:37.026418 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 267985us
I20250831 13:58:37.086372 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:37.087544 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.088124 20573 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:37.089231 22127 consensus_queue.cc:237] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:36.915333 20568 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:36.903386 22146 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.091619 22146 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.998399 22182 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.034551 20647 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 1.
I20250831 13:58:37.093389 20647 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:37.093904 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 983) took 1641 ms. Trace:
I20250831 13:58:37.094244 20500 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:37.094668 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.094998 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.095225 20495 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:37.095532 20496 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:37.094172 20647 rpcz_store.cc:276] 0831 13:58:35.452813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.092793 (+1639980us) service_pool.cc:225] Handling call
0831 13:58:37.093112 (+ 319us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.093704 (+ 592us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.093858 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:37.097437 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.097779 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 984) took 1644 ms. Trace:
I20250831 13:58:37.042178 20541 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.046463 20396 catalog_manager.cc:5582] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:37.098408 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.022992 20643 rpcz_store.cc:276] 0831 13:58:35.453846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.549334 (+1095488us) service_pool.cc:225] Handling call
0831 13:58:37.022251 (+472917us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540030edd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:37.022345 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":235516800}
I20250831 13:58:37.101506 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 716) took 1607 ms. Trace:
I20250831 13:58:37.101214 20396 catalog_manager.cc:5582] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 4, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 4 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:37.101831 20643 rpcz_store.cc:276] 0831 13:58:35.494229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.100666 (+1606437us) service_pool.cc:225] Handling call
0831 13:58:37.100970 (+ 304us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.101172 (+ 202us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:37.101195 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:37.101250 (+ 55us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:37.101314 (+ 64us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.101364 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:37.101458 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:37.050787 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.049513 20642 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:37.052017 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 100251us
I20250831 13:58:37.103168 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.104844 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.105778 21609 consensus_queue.cc:1046] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:37.052960 22152 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:37.053740 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 103804us
I20250831 13:58:37.107501 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:36.983918 22080 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:37.135471 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 93392.5us
I20250831 13:58:37.027515 22210 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:37.061378 20569 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:37.030902 22278 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:37.062633 21209 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.064971 21208 raft_consensus.cc:1238] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:37.065057 22238 consensus_queue.cc:237] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.074029 22125 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.061578 22066 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.076043 22157 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:36.917508 21174 leader_election.cc:400] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7.
I20250831 13:58:37.077283 21588 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.077809 22216 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:37.078044 20466 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.170430 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:37.085700 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.052551 20646 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:37.083062 20651 rpcz_store.cc:276] 0831 13:58:35.452156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.030700 (+1578544us) service_pool.cc:225] Handling call
0831 13:58:37.044719 (+ 14019us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.066471 (+ 21752us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.082318 (+ 15847us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b5400372b54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:37.082458 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48752000}
I20250831 13:58:37.098106 20647 rpcz_store.cc:276] 0831 13:58:35.453011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.096759 (+1643748us) service_pool.cc:225] Handling call
0831 13:58:37.097177 (+ 418us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.097427 (+ 250us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:37.097454 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:37.097506 (+ 52us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:37.097540 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.097605 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:37.097722 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:37.144573 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.094091 21659 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:36.815110 22149 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.157370 20645 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:37.159723 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.174371 20569 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:37.167052 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:37.167685 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.167591 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.168679 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.177655 20573 raft_consensus.cc:2391] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:37.169410 20617 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:37.171108 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.171383 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.178448 20500 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
W20250831 13:58:37.109367 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.160347 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:37.179239 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.155539 21609 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:37.180078 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 102128us
I20250831 13:58:37.180354 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.178404 20568 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:37.152108 22210 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.175074 20498 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:37.157248 22152 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.188898 20499 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:37.180451 20396 catalog_manager.cc:5582] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 3, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:37.186574 20617 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:37.192246 20643 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:37.193825 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 720) took 1600 ms. Trace:
I20250831 13:58:37.194237 20643 rpcz_store.cc:276] 0831 13:58:35.592951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.151132 (+1558181us) service_pool.cc:225] Handling call
0831 13:58:37.191946 (+ 40814us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.193556 (+ 1610us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.193767 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:37.191910 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.195798 20570 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:37.200286 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.201939 20570 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:37.208076 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:37.208554 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.210099 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.210350 20651 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250831 13:58:37.162564 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 245404us
I20250831 13:58:37.211050 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.159873 22067 consensus_queue.cc:1046] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20250831 13:58:37.211715 20573 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:37.210209 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.212786 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.189616 22210 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.187077 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 718) took 1692 ms. Trace:
I20250831 13:58:37.216383 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.166126 22066 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.216835 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.197129 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.219530 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.219638 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.219638 20466 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:37.220361 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.207898 22216 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.198141 22125 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:37.212914 22178 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:37.233891 20497 raft_consensus.cc:2408] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:37.191372 22221 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.237438 20541 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.213624 21659 raft_consensus.cc:695] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:37.238806 20574 raft_consensus.cc:2408] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:37.240142 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.237877 20568 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:37.210760 21708 consensus_queue.cc:1046] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20250831 13:58:37.242928 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:37.243660 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44520.8us
I20250831 13:58:37.245615 20616 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.246462 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:37.247123 20643 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:37.227313 22286 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.251371 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:37.251447 20396 catalog_manager.cc:5582] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 2 to 3, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:37.206844 22149 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.237428 21205 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:37.260118 20497 raft_consensus.cc:1273] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:37.270226 22100 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:37.287782 22067 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.240653 22182 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:37.335244 22067 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:37.335595 22182 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.336362 22149 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.220057 20642 rpcz_store.cc:276] 0831 13:58:35.494903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.103891 (+1608988us) service_pool.cc:225] Handling call
0831 13:58:37.104205 (+ 314us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.104427 (+ 222us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:37.135458 (+ 31031us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:37.154104 (+ 18646us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:37.158456 (+ 4352us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.169988 (+ 11532us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:37.181699 (+ 11711us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:37.337085 22067 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:37.270011 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 60000us
I20250831 13:58:37.269824 22220 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.289202 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:37.250367 20646 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:37.284650 20541 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.239660 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:37.242599 22157 consensus_queue.cc:1046] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20250831 13:58:37.256948 22216 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.285416 20494 raft_consensus.cc:2366] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:37.242547 22096 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:37.240067 22125 raft_consensus.cc:695] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:37.284823 21588 raft_consensus.cc:3053] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:37.283885 22281 raft_consensus.cc:3053] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:37.289399 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:37.270053 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:37.270846 22190 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:37.286973 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 710) took 2132 ms. Trace:
I20250831 13:58:37.291288 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:37.297328 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 48437.2us
I20250831 13:58:37.298249 20651 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
W20250831 13:58:37.301100 22307 log.cc:927] Time spent T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250831 13:58:37.300901 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:37.301968 22223 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 13:58:37.304692 22080 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.370065 22080 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.372426 22080 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.305406 22291 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.306968 22078 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:37.373348 22291 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.375456 22291 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:37.300741 22303 log.cc:927] Time spent T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20250831 13:58:37.312047 22286 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.314906 22278 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.377512 21208 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:37.378285 22286 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.379274 22278 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.380043 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:37.380756 20495 raft_consensus.cc:2391] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:37.381485 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.382606 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.383912 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:37.384989 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:37.385201 20672 debug-util.cc:398] Leaking SignalData structure 0x7b08001d11e0 after lost signal to thread 19369
W20250831 13:58:37.386365 20672 debug-util.cc:398] Leaking SignalData structure 0x7b08000ddea0 after lost signal to thread 20434
I20250831 13:58:37.317139 20500 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:37.317754 22270 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:37.387785 20672 debug-util.cc:398] Leaking SignalData structure 0x7b080036c5a0 after lost signal to thread 20526
W20250831 13:58:37.388094 20672 debug-util.cc:398] Leaking SignalData structure 0x7b080036c5e0 after lost signal to thread 20527
I20250831 13:58:37.388697 22270 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:37.320936 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:37.389254 20672 debug-util.cc:398] Leaking SignalData structure 0x7b0800372a80 after lost signal to thread 20601
W20250831 13:58:37.389612 20672 debug-util.cc:398] Leaking SignalData structure 0x7b0800372b20 after lost signal to thread 20602
W20250831 13:58:37.389787 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.389917 22125 consensus_queue.cc:237] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:37.391067 20672 debug-util.cc:398] Leaking SignalData structure 0x7b080036bf20 after lost signal to thread 20675
I20250831 13:58:37.321264 20493 raft_consensus.cc:2408] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:37.392307 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:37.393316 20672 debug-util.cc:398] Leaking SignalData structure 0x7b0800255060 after lost signal to thread 21234
I20250831 13:58:37.328200 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.323808 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.394496 20499 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:37.329018 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:37.395370 20645 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:37.389257 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 982) took 1936 ms. Trace:
I20250831 13:58:37.270542 22206 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:37.331283 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 715) took 2154 ms. Trace:
I20250831 13:58:37.334209 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.330895 22100 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.339457 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:37.344803 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 60234.4us
I20250831 13:58:37.350440 21588 raft_consensus.cc:738] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.349515 20569 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 2.
I20250831 13:58:37.339274 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:37.356050 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:37.341092 21609 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:37.427865 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.354204 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.363451 20466 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.357569 20644 rpcz_store.cc:276] 0831 13:58:35.154241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.013175 (+1858934us) service_pool.cc:225] Handling call
0831 13:58:37.013531 (+ 356us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:37.013913 (+ 382us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:37.013939 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:37.014398 (+ 459us) log.cc:844] Serialized 26 byte log entry
0831 13:58:37.181964 (+167566us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:37.182011 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.182073 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:37.286757 (+104684us) raft_consensus.cc:1683] finished
0831 13:58:37.286775 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:37.286898 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":95,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"thread_start_us":165057,"threads_started":1,"wal-append.queue_time_us":212964}
I20250831 13:58:37.343886 22067 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:37.347950 22096 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.364766 21708 raft_consensus.cc:2802] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:37.375589 20617 leader_election.cc:400] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:37.393476 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.331015 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:37.438580 20643 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:37.396649 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.439365 20500 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:37.399853 20651 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "ccee615469564d43800cb6f12f325e22"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
W20250831 13:58:37.347365 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 58268.8us
I20250831 13:58:37.386166 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.357365 22190 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.352114 22281 raft_consensus.cc:738] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.444958 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.445191 20500 raft_consensus.cc:1273] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:37.389281 22270 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:37.403167 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:37.424566 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 68611.8us
I20250831 13:58:37.426573 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.395784 20649 rpcz_store.cc:276] 0831 13:58:35.452608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.077385 (+1624777us) service_pool.cc:225] Handling call
0831 13:58:37.077744 (+ 359us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:37.078087 (+ 343us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:37.078113 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:37.078548 (+ 435us) log.cc:844] Serialized 26 byte log entry
0831 13:58:37.221095 (+142547us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:37.239424 (+ 18329us) log.cc:844] Serialized 14 byte log entry
0831 13:58:37.257912 (+ 18488us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.257979 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:37.388983 (+131004us) raft_consensus.cc:1683] finished
0831 13:58:37.389029 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:37.389186 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18251,"prepare.queue_time_us":25989,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":15752,"raft.queue_time_us":44907,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":141,"thread_start_us":58498,"threads_started":1,"wal-append.queue_time_us":266213}
I20250831 13:58:37.346876 22220 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.368423 22182 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.398535 20648 rpcz_store.cc:276] 0831 13:58:35.177336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.017824 (+1840488us) service_pool.cc:225] Handling call
0831 13:58:37.018545 (+ 721us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:37.018783 (+ 238us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:37.165156 (+146373us) log.cc:844] Serialized 14 byte log entry
0831 13:58:37.330895 (+165739us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:37.330958 (+ 63us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:37.331020 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.331101 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:37.331224 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":35548,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":29,"raft.queue_time_us":1227,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":74,"thread_start_us":278091,"threads_started":2,"wal-append.queue_time_us":193070}
I20250831 13:58:37.405103 22100 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:37.433416 21609 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:37.424474 22286 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.404089 22066 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.399605 21210 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:37.436064 22221 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.460950 22238 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:37.444911 22190 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:37.476240 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:37.472306 22209 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:37.476349 22238 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.448997 22127 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:37.435458 22096 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.402348 22202 consensus_queue.cc:1046] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
W20250831 13:58:37.478065 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 102676us
I20250831 13:58:37.466259 20498 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:37.470925 20496 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:37.475828 20499 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:37.464762 20649 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:37.404261 21659 consensus_queue.cc:237] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.482676 22238 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:37.511112 21914 raft_consensus.cc:3053] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:37.519284 21914 raft_consensus.cc:738] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.499490 20497 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:37.482779 22127 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.520900 22127 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.520112 21914 consensus_queue.cc:260] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.504961 22196 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.435132 21708 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:37.524470 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 161244us
W20250831 13:58:37.482335 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:37.484074 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:37.532824 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.433754 20642 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:37.534703 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.530346 20645 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:37.539845 20643 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
W20250831 13:58:37.499066 22310 log.cc:927] Time spent T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20250831 13:58:37.441416 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 184263us
W20250831 13:58:37.525272 22313 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
W20250831 13:58:37.525517 22306 log.cc:927] Time spent T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20250831 13:58:37.482298 20467 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:37.474270 22270 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:37.590014 22196 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.535595 21708 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:37.590754 21941 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.234s
I20250831 13:58:37.561131 20651 raft_consensus.cc:604] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.592996 20651 raft_consensus.cc:612] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:58:37.593189 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46464 (request call id 37) took 2141 ms. Trace:
I20250831 13:58:37.594362 20651 raft_consensus.cc:1273] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250831 13:58:37.561504 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.562711 22229 consensus_queue.cc:1046] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20250831 13:58:37.565294 22152 raft_consensus.cc:3053] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:37.568363 20649 raft_consensus.cc:1273] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:37.569281 20617 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.481690 22209 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.586769 20501 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:37.498891 21208 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:37.593684 20650 rpcz_store.cc:276] 0831 13:58:35.451701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.020644 (+1568943us) service_pool.cc:225] Handling call
0831 13:58:37.020947 (+ 303us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:37.023401 (+ 2454us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:37.023439 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:37.036512 (+ 13073us) log.cc:844] Serialized 26 byte log entry
0831 13:58:37.080690 (+ 44178us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b1800472a80. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171 00007fb9bc94909f
0831 13:58:37.208297 (+127607us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:37.208332 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.208396 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:37.592815 (+384419us) raft_consensus.cc:1683] finished
0831 13:58:37.592837 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:37.593103 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":398,"raft.queue_time_us":4032,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":8898,"spinlock_wait_cycles":29354752,"thread_start_us":127443,"threads_started":1,"wal-append.queue_time_us":302168}
I20250831 13:58:37.522709 21914 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:37.464908 22220 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.607232 20617 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.483711 22286 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.609014 20643 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:37.617777 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250831 13:58:37.602447 22152 raft_consensus.cc:738] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:37.621388 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:58:37.621681 22152 consensus_queue.cc:260] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:37.624004 22291 raft_consensus.cc:3053] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:37.624660 22291 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:37.626055 22270 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.626848 22182 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:37.627023 20651 raft_consensus.cc:1273] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:37.627432 22182 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:37.627996 22182 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:37.629251 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:37.531875 22100 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.631808 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:37.633500 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:58:37.570706 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:58:37.635066 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 81729.7us
W20250831 13:58:37.635680 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.614385 22206 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:37.636515 22206 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:37.625783 22291 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.601604 22209 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.637283 20649 raft_consensus.cc:1273] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:37.637833 20617 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:37.645601 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 163365us
I20250831 13:58:37.638355 20543 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:58:37.637029 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 233938us
I20250831 13:58:37.652882 20617 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:37.654208 20648 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:37.654318 20467 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:37.654742 20617 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.656414 20617 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:37.659507 20617 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:37.660713 20500 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:37.636914 22206 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:37.664453 20467 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:37.462965 22281 consensus_queue.cc:260] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.473783 21588 consensus_queue.cc:260] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.638834 22220 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:37.603329 22238 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.651682 22157 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:37.654153 22196 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:37.664445 22178 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:37.677304 22281 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:37.715204 22210 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:37.717257 22210 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:37.678068 21588 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:37.666968 22188 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:37.666222 21941 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:37.718958 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:37.719570 20568 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.719734 22286 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.651335 22216 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:37.707386 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 47983.9us
I20250831 13:58:37.722276 20541 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.723160 20541 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.723852 22159 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:37.723949 20541 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.724392 22159 raft_consensus.cc:695] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.658460 22152 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:37.657428 22080 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:37.719117 21941 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:37.725414 21941 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:37.725308 22159 consensus_queue.cc:237] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.670225 22202 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20250831 13:58:37.727336 21708 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.679471 22149 consensus_queue.cc:1046] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:37.728822 22149 raft_consensus.cc:3053] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:37.729182 22149 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:37.729297 21708 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:37.729439 22321 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.072s user 0.000s sys 0.003s
I20250831 13:58:37.730078 20493 raft_consensus.cc:2391] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:37.731952 22127 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.733132 22100 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.733863 22127 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.718958 22188 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:37.734766 22323 log.cc:927] Time spent T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20250831 13:58:37.734874 22188 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:37.736404 22096 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:37.737689 22238 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.687091 20543 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:37.730019 22149 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.648047 21609 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.739395 22149 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:37.741571 21609 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:37.743319 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.744280 20496 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:37.745800 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.665202 22301 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:37.671851 22291 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:37.751467 22066 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 13:58:37.752614 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 65646.4us
I20250831 13:58:37.753175 22238 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:37.753876 21588 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:37.754910 22078 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:37.755887 22100 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:37.756587 20541 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.759997 22178 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.777747 22078 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.708035 22157 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.761883 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:37.763247 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:37.763610 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:37.764988 20572 raft_consensus.cc:1273] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:37.767624 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 992) took 1087 ms. Trace:
I20250831 13:58:37.709081 22220 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.670598 22221 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:37.650969 22190 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.714025 22196 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.779453 20543 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:37.792320 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:37.780364 21588 raft_consensus.cc:695] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.747493 22096 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:37.746047 22286 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:37.781958 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:37.781913 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:37.748217 22301 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:37.784502 21708 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:37.798728 20496 raft_consensus.cc:1273] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:37.812000 20500 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:37.785676 20647 rpcz_store.cc:276] 0831 13:58:36.680505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.183205 (+502700us) service_pool.cc:225] Handling call
0831 13:58:37.207717 (+ 24512us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:37.243630 (+ 35913us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:37.243659 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:37.262220 (+ 18561us) log.cc:844] Serialized 26 byte log entry
0831 13:58:37.579075 (+316855us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:37.579113 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.579178 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:37.767398 (+188220us) raft_consensus.cc:1683] finished
0831 13:58:37.767420 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:37.767551 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19276,"raft.queue_time_us":32865,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":21113,"thread_start_us":236649,"threads_started":1,"wal-append.queue_time_us":427760}
I20250831 13:58:37.783265 22078 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:37.801864 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.785683 22157 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:37.797230 20568 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:37.810690 20569 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:37.820724 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:37.813562 21588 consensus_queue.cc:237] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.761502 22188 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.790254 22220 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.816437 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:37.797180 21210 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:37.781526 22278 consensus_queue.cc:1035] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20250831 13:58:37.824554 20501 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:37.791415 22221 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:37.798148 22196 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:37.799376 22216 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:37.803027 22066 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:37.784459 22067 raft_consensus.cc:695] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.781441 22178 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.776782 21941 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.861418 20499 raft_consensus.cc:1273] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:37.810612 22301 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:37.816938 22238 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:37.864360 22221 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:37.860284 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:37.899015 22217 raft_consensus.cc:2762] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20250831 13:58:37.899530 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.899132 21207 raft_consensus.cc:2953] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 2 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:37.899159 22188 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.829658 22281 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20250831 13:58:37.909228 22301 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.912684 20642 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 2.
I20250831 13:58:37.913982 22067 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:37.915405 22330 consensus_queue.cc:1035] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:37.915555 22182 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.886085 22216 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.931993 22190 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.933512 22178 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:37.937829 22196 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.937968 22209 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:37.968014 22229 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:37.941262 21941 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:37.943943 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 59221.2us
W20250831 13:58:37.943290 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:37.887562 22327 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20250831 13:58:37.961848 22238 raft_consensus.cc:695] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:37.965883 22286 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:37.938724 22221 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:37.969805 20493 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:38.003335 22100 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:38.005960 22238 consensus_queue.cc:237] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:38.014021 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:38.014981 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:38.020179 20495 raft_consensus.cc:2366] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
W20250831 13:58:38.040125 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:38.039973 22182 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.041800 22329 consensus_queue.cc:1046] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:38.043572 22329 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:38.045230 22329 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:38.047211 22221 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.047974 22281 raft_consensus.cc:3053] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:38.048357 22281 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:38.049259 22281 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.050355 22281 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:38.044920 22220 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:38.055518 20572 raft_consensus.cc:2391] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:38.059788 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:38.056047 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 379831us
W20250831 13:58:38.063921 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 404495us
I20250831 13:58:38.065711 20617 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.067338 20617 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.010828 22286 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.077232 20617 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.083979 20617 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.086225 20617 leader_election.cc:400] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:38.086779 20617 leader_election.cc:403] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:38.070883 22182 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:38.101979 21659 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 13:58:38.135739 20497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.007s sys 0.046s
W20250831 13:58:38.128506 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:38.142613 20497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.007s sys 0.050s
I20250831 13:58:38.141911 20617 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:38.101608 22190 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:38.140889 22178 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 5
I20250831 13:58:38.146333 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 726) took 1318 ms. Trace:
I20250831 13:58:38.146754 20644 rpcz_store.cc:276] 0831 13:58:36.828313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.429414 (+601101us) service_pool.cc:225] Handling call
0831 13:58:37.443352 (+ 13938us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:37.445691 (+ 2339us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:37.445722 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:37.463985 (+ 18263us) log.cc:844] Serialized 26 byte log entry
0831 13:58:37.811493 (+347508us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:37.811535 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:37.811591 (+ 56us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:38.146090 (+334499us) raft_consensus.cc:1683] finished
0831 13:58:38.146111 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:38.146273 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7239,"raft.queue_time_us":2336,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11072,"thread_start_us":333759,"threads_started":1,"wal-append.queue_time_us":613828}
I20250831 13:58:38.148512 22338 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:38.148938 22338 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:38.149309 22338 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:38.151752 22286 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.152854 22301 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.152577 22157 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:38.154764 22216 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.140731 22206 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.155516 22178 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:38.157079 22206 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.158565 20650 raft_consensus.cc:1238] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:38.158483 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.076530 22291 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:38.159296 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:38.158725 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 343761us
I20250831 13:58:38.167834 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:38.177030 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.175853 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:38.183256 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.209695 20498 raft_consensus.cc:2408] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:38.182875 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.182966 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.209107 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:38.189896 20616 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.169602 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:38.212396 20495 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:38.212844 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 734) took 1382 ms. Trace:
I20250831 13:58:38.213272 20650 rpcz_store.cc:276] 0831 13:58:36.830335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.604887 (+774552us) service_pool.cc:225] Handling call
0831 13:58:37.627162 (+ 22275us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:38.162084 (+534922us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:38.212628 (+ 50544us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b540030fcd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:38.212791 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1221652864}
I20250831 13:58:38.216768 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.216861 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.217633 20643 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:38.218559 20616 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.221534 20497 raft_consensus.cc:2366] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:38.217674 20499 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:38.222759 20617 leader_election.cc:400] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:38.223192 20617 leader_election.cc:403] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20250831 13:58:38.219918 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.222220 20616 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.142405 21659 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.227504 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.225436 20496 raft_consensus.cc:2408] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:38.225263 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.162659 21708 raft_consensus.cc:695] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:38.245599 20647 raft_consensus.cc:1273] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:58:38.231787 20650 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:38.246161 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 435800us
I20250831 13:58:38.247980 20644 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 13:58:38.124827 22078 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.249336 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.249632 20501 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:38.231855 20651 raft_consensus.cc:3053] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:38.249367 20617 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.175762 22178 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.251848 20616 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:38.252450 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:38.245079 21659 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:38.253650 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.175510 22157 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:38.270493 20497 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:38.274888 22186 consensus_queue.cc:1035] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:38.276736 22291 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:38.277734 20617 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.278810 20467 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.279129 22210 consensus_queue.cc:1046] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:38.280241 20500 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250831 13:58:38.278561 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.281093 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:38.255426 22100 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:38.281745 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.283419 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.283452 20616 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.284442 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.314754 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.270823 20651 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:38.327939 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.333240 20651 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.183442 20642 raft_consensus.cc:1273] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:38.188619 22196 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.286191 20650 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:38.287189 20494 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:38.290088 21588 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:38.347867 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 335180us
I20250831 13:58:38.348838 21588 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.226104 22336 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:38.182663 22159 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20250831 13:58:38.294317 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.295508 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:38.440361 21588 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.440569 20501 raft_consensus.cc:2408] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 13:58:38.300143 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.303051 20571 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:38.441165 20568 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:38.229740 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.304100 20570 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:38.281455 22209 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.285171 22182 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:38.282752 22301 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:38.315869 20648 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:38.328213 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.328536 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.293293 21609 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.293555 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.228497 21914 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20250831 13:58:38.331547 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.332020 20616 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.315053 20643 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:38.312790 20649 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:38.334430 20651 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:38.335582 20572 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:38.335891 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.452286 20576 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:38.452518 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:38.452699 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 120809us
I20250831 13:58:38.294006 22220 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:38.301191 22291 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.296187 22229 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:38.309209 22206 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:38.353796 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.353942 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.354923 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.459468 20573 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:38.348407 22286 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:58:38.376507 22339 log.cc:927] Time spent T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20250831 13:58:38.391433 22341 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250831 13:58:38.331604 22100 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.474346 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.289489 22178 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.475427 20650 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:38.475579 22100 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:38.421192 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 71572us
I20250831 13:58:38.432796 20499 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:38.305507 22066 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:38.470552 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.375154 22342 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:38.461777 22206 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.482061 22206 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:38.482700 22078 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.483665 22152 raft_consensus.cc:3053] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:38.484089 22152 raft_consensus.cc:738] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:38.484892 22152 consensus_queue.cc:260] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.486164 22152 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:38.486788 22066 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.488740 22066 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.491227 20569 raft_consensus.cc:2408] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:38.461257 22209 consensus_queue.cc:1035] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
W20250831 13:58:38.495791 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:38.481937 22342 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.458825 22291 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:38.438376 20497 raft_consensus.cc:1273] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:38.439846 20647 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
W20250831 13:58:38.439395 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:38.450320 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.228297 21175 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:38.453429 20568 raft_consensus.cc:2391] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:38.453464 20616 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:38.438926 21992 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.439651 22157 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:38.458880 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.476485 22178 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.464062 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.467087 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.468593 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.469482 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.469610 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.477767 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:38.477787 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.478940 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.397380 21708 consensus_queue.cc:237] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.456187 22190 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:38.309324 22216 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:38.497578 20650 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:38.496807 20495 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
W20250831 13:58:38.506721 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 53342.5us
I20250831 13:58:38.506273 20651 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250831 13:58:38.506189 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.511230 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.460996 20467 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:38.481341 21210 consensus_meta.cc:287] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.603s user 0.008s sys 0.002s
W20250831 13:58:38.591908 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:38.458629 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 124597us
I20250831 13:58:38.592700 20651 pending_rounds.cc:85] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Aborting all ops after (but not including) 2
I20250831 13:58:38.593287 20617 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:38.594172 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 133279us
I20250831 13:58:38.594245 20577 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 5
W20250831 13:58:38.554870 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 75194.1us
I20250831 13:58:38.593583 20568 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:38.595245 20575 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:38.454111 21609 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.596125 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.596022 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.520020 20543 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.596197 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.609205 20646 raft_consensus.cc:2391] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:38.593111 20498 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:38.499163 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.610325 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.604276 20651 pending_rounds.cc:107] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250831 13:58:38.610739 20647 raft_consensus.cc:2391] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:38.610929 20651 raft_consensus.cc:2887] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:38.507028 22342 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:38.615036 20541 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.615166 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.615950 20643 raft_consensus.cc:2391] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:38.603322 22322 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:38.551028 22337 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:38.612967 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.513242 22229 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:38.603533 22096 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:38.617587 20541 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:38.549929 22301 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:38.621836 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 181752us
W20250831 13:58:38.642846 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:38.643527 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:38.623298 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.610219 22327 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:38.624329 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.644797 20650 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:38.551384 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 333425us
I20250831 13:58:38.606459 22216 raft_consensus.cc:695] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:38.602252 21210 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:38.629679 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:38.630820 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 110924us
I20250831 13:58:38.512051 22286 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.102s
I20250831 13:58:38.554028 22078 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20250831 13:58:38.633060 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.711732 20647 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:38.631924 20651 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:38.549656 21992 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.519099 22100 consensus_queue.cc:1046] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
I20250831 13:58:38.614725 22178 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.641258 20573 raft_consensus.cc:1238] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:38.595711 22157 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:38.629051 22281 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.632705 22347 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:38.634881 21914 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:38.646811 22301 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:38.647794 22354 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20250831 13:58:38.647346 22338 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:38.648836 20577 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:38.649556 20466 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:38.630216 22182 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 2. Current term is 3.
I20250831 13:58:38.627542 22188 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:38.650713 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.741600 20643 raft_consensus.cc:2391] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:38.633853 22190 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:38.664156 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46464 (request call id 40) took 1096 ms. Trace:
I20250831 13:58:38.636751 22337 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.746762 20646 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:38.745749 20645 rpcz_store.cc:276] 0831 13:58:37.567668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.636098 (+ 68430us) service_pool.cc:225] Handling call
0831 13:58:37.654019 (+ 17921us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:38.057763 (+403744us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:38.057808 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:38.071984 (+ 14176us) log.cc:844] Serialized 26 byte log entry
0831 13:58:38.425436 (+353452us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:38.425491 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:38.425577 (+ 86us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:38.663961 (+238384us) raft_consensus.cc:1683] finished
0831 13:58:38.663984 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:38.664104 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5032,"raft.queue_time_us":80502,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":17355,"thread_start_us":656761,"threads_started":2,"wal-append.queue_time_us":391618}
I20250831 13:58:38.747912 22337 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.748418 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.749387 20645 raft_consensus.cc:2391] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:38.666184 20574 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:38.667969 20468 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:38.656466 22327 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.675985 22159 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:38.676599 20568 raft_consensus.cc:1273] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:38.676826 22216 consensus_queue.cc:237] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.628911 22278 consensus_queue.cc:1046] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20250831 13:58:38.681717 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:38.757716 20617 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.764556 20569 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:38.683842 20576 raft_consensus.cc:1273] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:38.688022 20543 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:58:38.768147 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 80261.4us
I20250831 13:58:38.691419 22329 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:38.696588 22067 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:38.698601 21609 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:38.698865 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:38.691745 22353 log.cc:927] Time spent T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.141s user 0.002s sys 0.001s
I20250831 13:58:38.637115 22096 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:38.644167 22220 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:38.705515 20572 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:38.709352 20575 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:38.645938 20642 raft_consensus.cc:2391] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:38.716369 20541 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:38.635594 22336 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:38.722339 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.796145 20651 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:38.724689 20650 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
W20250831 13:58:38.725498 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:38.797394 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:38.738272 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.797638 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.744616 20644 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:38.750320 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:38.796911 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 80644.2us
I20250831 13:58:38.801304 20541 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:38.755450 20646 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:58:38.756063 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 106555us
W20250831 13:58:38.756148 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 88303.5us
I20250831 13:58:38.731099 22178 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.733062 21914 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.754601 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.737794 22066 raft_consensus.cc:3053] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:38.785320 20574 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20250831 13:58:38.748436 22337 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.784076 20543 leader_election.cc:400] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:38.788254 20575 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 13:58:38.819069 20543 leader_election.cc:403] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:38.781106 21175 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:38.789003 22358 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:38.789361 21992 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.789427 22286 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:38.783541 22356 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
W20250831 13:58:38.795215 22346 log.cc:927] Time spent T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20250831 13:58:38.803280 20466 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:38.837201 20466 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:38.838295 20466 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:38.732987 22281 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.808101 22229 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:38.804785 20571 raft_consensus.cc:1273] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:58:38.805377 20568 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:38.763998 22149 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:38.815811 22066 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:38.754153 22357 consensus_queue.cc:1046] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
I20250831 13:58:38.813491 21914 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:38.763928 22190 raft_consensus.cc:695] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:38.815026 20649 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:38.784440 22347 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:38.819190 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.820518 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:38.821200 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:38.783524 22210 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.787966 22096 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:38.787765 22100 raft_consensus.cc:3053] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
W20250831 13:58:38.782707 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 151219us
I20250831 13:58:38.754971 22188 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.826985 20645 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:38.828634 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:38.830650 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 61679.6us
W20250831 13:58:38.803654 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 51162.1us
I20250831 13:58:38.814209 22078 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:38.803982 20616 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:38.886065 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 82144.3us
I20250831 13:58:38.809531 22336 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.887115 20616 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.859481 22347 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.887934 22347 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:38.888439 20616 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.808390 22322 raft_consensus.cc:695] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 2 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:38.889856 22322 consensus_queue.cc:237] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:38.865952 22206 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.893304 22206 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:38.808252 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 70063.4us
I20250831 13:58:38.894523 20467 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.823503 22358 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.821115 21609 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.783706 22159 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.787549 22067 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:38.847447 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:38.849705 20651 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:38.818406 22220 raft_consensus.cc:695] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:38.850461 20577 raft_consensus.cc:1238] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:38.903729 20647 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:38.841881 21992 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:38.841398 22221 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:38.844408 22196 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:38.865087 20498 raft_consensus.cc:2391] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:38.845490 22356 raft_consensus.cc:695] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:38.844225 22278 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5.
I20250831 13:58:38.844460 22157 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:38.825335 22337 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.874676 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.937112 20644 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:38.902947 22362 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:38.877725 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:38.878942 22301 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.879570 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:38.879689 22359 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.839510 22286 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.965174 22359 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.966446 22286 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.967083 22359 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.883636 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.967971 20644 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:38.968261 20645 raft_consensus.cc:2408] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:38.968761 21210 raft_consensus.cc:2391] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:38.884372 21659 consensus_meta.cc:287] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.569s user 0.009s sys 0.001s
I20250831 13:58:38.969806 22301 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:38.970214 22301 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.970660 22301 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:38.970247 21659 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.972859 21659 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.875108 22182 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.974790 22182 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.975770 22188 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.977260 22182 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.969372 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.884867 21208 raft_consensus.cc:1238] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:38.868451 22100 raft_consensus.cc:738] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:38.979579 22338 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:38.980103 22100 consensus_queue.cc:260] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.981417 22100 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:38.981189 21914 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.850802 22360 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.923663 21176 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.983032 22360 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:38.983471 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 81150.7us
I20250831 13:58:38.868597 22210 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.985707 22362 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:38.985828 22210 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.818081 22209 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:38.814582 22238 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:38.986851 22238 raft_consensus.cc:695] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:38.988005 22238 consensus_queue.cc:237] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.989861 22360 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.839913 20466 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:38.991243 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 151419us
I20250831 13:58:38.992314 20466 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:38.897734 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.993748 20646 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:38.993956 22210 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:38.994822 20646 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:38.996062 20646 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [3.2-4.3]
I20250831 13:58:38.996727 21588 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.903398 20467 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:38.997159 22357 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:38.986812 22209 raft_consensus.cc:695] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:38.856556 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:38.997496 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 94211.9us
I20250831 13:58:38.997712 21992 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:38.998056 20649 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:38.864521 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:38.998451 21992 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:38.998983 20642 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:38.997191 21588 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:38.997763 22329 consensus_queue.cc:1046] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20250831 13:58:39.001968 21588 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:39.002417 21588 raft_consensus.cc:3053] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:38.885357 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:38.927570 20541 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:38.852558 22361 consensus_queue.cc:1046] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20250831 13:58:38.909685 20647 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:38.929278 20616 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:38.885582 22078 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:39.014535 22078 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:38.902468 22358 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:39.016249 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 154501us
I20250831 13:58:38.907239 22335 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:38.943683 21207 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:38.946831 22369 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:38.923414 22221 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:38.947378 22281 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.937939 22066 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.948740 22067 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.906661 21609 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.928966 22343 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20250831 13:58:38.929554 22327 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.950829 22365 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:39.084765 22365 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.085278 22365 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:38.942385 22196 raft_consensus.cc:695] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:38.956136 22190 consensus_queue.cc:237] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.957127 22356 consensus_queue.cc:237] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:38.962204 22220 consensus_queue.cc:237] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:38.962956 22159 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:38.963327 22178 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.964669 22337 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.982434 22202 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.986956 22372 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:38.864022 21941 raft_consensus.cc:3053] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:38.994450 22210 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:38.910816 20651 raft_consensus.cc:1273] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:38.999114 20644 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:39.000264 20642 raft_consensus.cc:1238] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:38.888401 22336 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:38.851444 22229 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:39.007380 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 78172.2us
I20250831 13:58:38.978482 20645 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:39.008132 21588 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:38.882532 20650 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
W20250831 13:58:39.009673 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 82195.6us
W20250831 13:58:39.109601 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 116291us
I20250831 13:58:39.109691 20616 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.111375 20650 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:39.112453 20466 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:39.009727 20649 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:58:39.112900 20616 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.113703 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.113960 20466 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:39.114285 20649 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:39.114339 20616 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.019987 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.044034 20647 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 13:58:39.045578 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.115410 20502 raft_consensus.cc:2391] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:39.115473 20501 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:39.044716 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.035332 20398 catalog_manager.cc:5582] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ab7ade51fa134a1b814fab748f01d43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:58:39.049180 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.116386 20466 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:39.116608 20496 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:39.116822 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.117478 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.055611 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.118011 20496 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:39.118165 20617 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.118875 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.049013 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.015888 22152 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:39.110550 21588 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:39.056246 22291 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.075169 22149 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:39.046350 22221 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.124893 20497 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:39.125662 20541 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.127013 20541 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.105969 21941 raft_consensus.cc:738] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:39.123665 20616 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.128381 20645 raft_consensus.cc:1238] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 1. Current term is 2. Ops: []
W20250831 13:58:39.090585 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 92417.7us
I20250831 13:58:39.090713 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.046973 22374 raft_consensus.cc:2802] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:39.096474 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:39.039207 22096 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.090143 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.131433 20499 raft_consensus.cc:2391] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:39.130519 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.132211 20647 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:39.132949 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.132302 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.117974 20466 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:39.120600 20617 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.060916 22216 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.136718 22216 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.138679 22216 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.139209 22281 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:39.139492 21207 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:58:39.140029 20500 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.141101 22078 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.141873 22210 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:39.143790 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:39.121843 21588 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.029803 22359 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:39.121803 20644 raft_consensus.cc:1273] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:39.148747 20617 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.149871 20500 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:39.122749 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.100288 22335 raft_consensus.cc:695] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:39.121811 22343 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:39.124388 22369 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:39.123322 22337 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:39.127285 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:39.156069 22182 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:39.160041 22380 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:39.160398 22182 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.160696 22380 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.161314 22380 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.161525 22378 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:39.162200 22378 raft_consensus.cc:695] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:39.133527 22067 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250831 13:58:39.163163 22378 consensus_queue.cc:237] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.139581 22281 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:39.164412 22281 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:39.136065 21708 consensus_queue.cc:1046] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:39.166105 21708 raft_consensus.cc:3053] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:39.166452 21708 raft_consensus.cc:738] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:39.157521 22188 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:39.167577 22188 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.167273 21708 consensus_queue.cc:260] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.168082 22188 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.168576 21708 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.125036 22338 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:39.138736 22375 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:39.146857 22336 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:39.106132 22202 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.119567 22178 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:39.154987 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.188711 20645 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:39.127444 22152 raft_consensus.cc:695] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:39.127444 22286 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20250831 13:58:39.160889 22182 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:39.093536 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 335033us
I20250831 13:58:39.087218 22209 consensus_queue.cc:237] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:39.131817 22327 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:39.176286 20497 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:39.134101 20541 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.146615 20493 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 4
W20250831 13:58:39.179673 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 61783.7us
I20250831 13:58:39.150823 22374 raft_consensus.cc:695] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:39.107374 22372 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.156268 22347 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.134071 22159 consensus_queue.cc:1046] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20250831 13:58:39.240187 20466 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:39.171480 21914 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.157475 22238 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:39.261546 20649 raft_consensus.cc:1273] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20250831 13:58:39.253440 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 47367.5us
I20250831 13:58:39.275358 20617 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.169710 22338 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.186988 22149 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:39.272756 22343 consensus_queue.cc:1046] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20250831 13:58:39.282128 22360 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:39.283979 22383 log.cc:927] Time spent T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20250831 13:58:39.184657 21941 consensus_queue.cc:260] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.289072 22329 raft_consensus.cc:3053] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:39.293260 22159 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election won for term 8
W20250831 13:58:39.293658 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 159618us
I20250831 13:58:39.295243 22357 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:39.295619 22096 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.204913 22078 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:39.298799 22370 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.164s user 0.001s sys 0.000s
I20250831 13:58:39.238824 21609 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:39.324028 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 194687us
I20250831 13:58:39.246371 22336 raft_consensus.cc:695] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:39.323662 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.240478 22327 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:39.276072 22378 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:39.157904 22196 consensus_queue.cc:237] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:39.337272 22377 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.249s user 0.002s sys 0.000s
I20250831 13:58:39.337457 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:39.338238 21205 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:39.339071 22374 consensus_queue.cc:237] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.340535 22152 consensus_queue.cc:237] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.326812 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.310537 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 762) took 1023 ms. Trace:
I20250831 13:58:39.302423 22337 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:39.303202 22149 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.317305 22238 raft_consensus.cc:695] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:39.361557 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1026) took 1085 ms. Trace:
W20250831 13:58:39.332083 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 51825.4us
I20250831 13:58:39.334321 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.363112 20501 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:39.318756 22356 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.260170 22210 consensus_queue.cc:1046] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20250831 13:58:39.329250 22159 raft_consensus.cc:695] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:39.333392 22336 consensus_queue.cc:237] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.304412 22338 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.328449 21992 raft_consensus.cc:3053] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:39.344705 22202 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.345284 22291 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:39.353880 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 211371us
I20250831 13:58:39.318447 22329 raft_consensus.cc:738] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:39.343725 20541 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.358287 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1017) took 1084 ms. Trace:
I20250831 13:58:39.394732 20651 rpcz_store.cc:276] 0831 13:58:38.273727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.110561 (+836834us) service_pool.cc:225] Handling call
0831 13:58:39.126010 (+ 15449us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:39.126322 (+ 312us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:39.126351 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:39.126801 (+ 450us) log.cc:844] Serialized 26 byte log entry
0831 13:58:39.299282 (+172481us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:39.356729 (+ 57447us) log.cc:844] Serialized 14 byte log entry
0831 13:58:39.357989 (+ 1260us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.358067 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:39.358109 (+ 42us) raft_consensus.cc:1683] finished
0831 13:58:39.358121 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:39.358236 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":600,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":181,"thread_start_us":223899,"threads_started":2,"wal-append.queue_time_us":193595}
I20250831 13:58:39.328109 21941 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:39.362651 20650 rpcz_store.cc:276] 0831 13:58:38.276108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.112040 (+835932us) service_pool.cc:225] Handling call
0831 13:58:39.128475 (+ 16435us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:39.128798 (+ 323us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:39.128826 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:39.133252 (+ 4426us) log.cc:844] Serialized 26 byte log entry
0831 13:58:39.320021 (+186769us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:39.360811 (+ 40790us) log.cc:844] Serialized 14 byte log entry
0831 13:58:39.361143 (+ 332us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.361254 (+ 111us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:39.361323 (+ 69us) raft_consensus.cc:1683] finished
0831 13:58:39.361341 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:39.361491 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1244,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":22302,"thread_start_us":173647,"threads_started":2,"wal-append.queue_time_us":178491}
I20250831 13:58:39.310570 22360 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.338495 22327 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:39.356397 22342 consensus_meta.cc:287] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.722s user 0.010s sys 0.000s
I20250831 13:58:39.352368 21609 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:39.360293 22149 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:39.387387 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1028) took 1110 ms. Trace:
W20250831 13:58:39.390481 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:39.399694 22387 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
W20250831 13:58:39.414455 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 70838.6us
I20250831 13:58:39.415572 21588 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:39.403321 22385 log.cc:927] Time spent T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20250831 13:58:39.410313 20647 rpcz_store.cc:276] 0831 13:58:38.276637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.149172 (+872535us) service_pool.cc:225] Handling call
0831 13:58:39.157977 (+ 8805us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:39.158298 (+ 321us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:39.158325 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:39.158759 (+ 434us) log.cc:844] Serialized 26 byte log entry
0831 13:58:39.325312 (+166553us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:39.386824 (+ 61512us) log.cc:844] Serialized 14 byte log entry
0831 13:58:39.387079 (+ 255us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.387144 (+ 65us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:39.387189 (+ 45us) raft_consensus.cc:1683] finished
0831 13:58:39.387201 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:39.387333 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10142,"raft.queue_time_us":3093,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":11850,"thread_start_us":109111,"threads_started":2,"wal-append.queue_time_us":172899}
I20250831 13:58:39.416210 21588 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.365386 22337 raft_consensus.cc:695] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:39.419678 22221 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.384740 21992 raft_consensus.cc:738] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:39.421567 20650 raft_consensus.cc:1238] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:39.420630 21205 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:39.434566 22372 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.426381 22188 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.302047 22335 consensus_queue.cc:237] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.437284 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.439327 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.404529 22229 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.407392 21609 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.379087 22356 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.440637 20500 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:58:39.446732 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 83199.8us
I20250831 13:58:39.440771 20617 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:39.355549 20642 rpcz_store.cc:276] 0831 13:58:38.287488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.191246 (+903758us) service_pool.cc:225] Handling call
0831 13:58:39.194924 (+ 3678us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:39.195132 (+ 208us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:39.195169 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:39.245205 (+ 50036us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:39.255813 (+ 10608us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.275779 (+ 19966us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:39.304918 (+ 29139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:39.428499 22157 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:39.407645 22202 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250831 13:58:39.388665 22358 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.444741 21588 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.453722 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.473065 20650 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:39.455065 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.471642 20617 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.441129 20573 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:39.475829 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.476850 20569 raft_consensus.cc:2366] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
W20250831 13:58:39.479642 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 112418us
I20250831 13:58:39.446341 22067 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.439939 22278 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.483918 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.485975 20573 raft_consensus.cc:2391] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:39.452703 21609 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:39.487167 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.487382 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:39.491093 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.488973 20646 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250831 13:58:39.489254 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.409168 22342 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.492367 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.493256 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:39.493530 20643 raft_consensus.cc:2366] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5.
I20250831 13:58:39.495030 20650 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:39.495275 20572 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:39.496289 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.496757 20569 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:39.445797 22238 consensus_queue.cc:237] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.502959 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:39.503151 22371 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.503576 22371 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.504525 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.506911 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.507516 20500 raft_consensus.cc:2408] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:58:39.508808 20616 leader_election.cc:400] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:39.509114 20616 leader_election.cc:403] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:39.480721 22229 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 13:58:39.485320 22301 consensus_meta.cc:287] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.514s user 0.005s sys 0.005s
I20250831 13:58:39.478749 22347 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.517748 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.518070 20648 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 3: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:39.518983 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:39.519343 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.505354 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.502460 20493 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:39.475584 22157 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:39.521238 22388 consensus_queue.cc:1046] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.143s
I20250831 13:58:39.521648 22221 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.521827 22157 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:39.522080 22388 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:39.490502 22078 consensus_queue.cc:1046] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20250831 13:58:39.451077 22096 consensus_queue.cc:1046] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20250831 13:58:39.523558 22100 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.493003 22343 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20250831 13:58:39.520874 22229 raft_consensus.cc:695] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:39.512432 22369 raft_consensus.cc:2762] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250831 13:58:39.521088 22371 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.522187 22221 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:39.534032 22337 consensus_queue.cc:237] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.546257 22221 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.548156 22221 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.522651 22388 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.514978 20541 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.555217 20617 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.516917 22387 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:39.530397 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.513751 22372 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:39.516498 22190 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.536406 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1038) took 1187 ms. Trace:
I20250831 13:58:39.537912 20572 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:39.516363 22361 raft_consensus.cc:3053] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:39.505548 22278 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:39.492311 22067 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.480399 22159 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:39.524041 22100 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.524529 22301 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.525465 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.512933 21992 consensus_queue.cc:260] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.561064 20501 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
W20250831 13:58:39.565001 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:39.563657 20651 rpcz_store.cc:276] 0831 13:58:38.349111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.396774 (+1047663us) service_pool.cc:225] Handling call
0831 13:58:39.424888 (+ 28114us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:39.425078 (+ 190us) raft_consensus.cc:1532] Early marking committed up to index 2
0831 13:58:39.435426 (+ 10348us) log.cc:844] Serialized 14 byte log entry
0831 13:58:39.536099 (+100673us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:39.536154 (+ 55us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:39.536190 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.536251 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:39.536351 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10000,"prepare.queue_time_us":26773,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":85,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":1729,"thread_start_us":99261,"threads_started":1}
I20250831 13:58:39.583388 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:39.559967 22395 raft_consensus.cc:3053] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:39.535410 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:39.583936 20617 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.583715 22149 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.545751 21207 raft_consensus.cc:1238] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:39.561066 22347 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:39.585011 22347 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.585393 22347 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:39.589567 22157 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.578100 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.591434 22149 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.584290 22395 raft_consensus.cc:738] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:39.562029 22372 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.472249 22329 consensus_queue.cc:260] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.560575 22371 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:39.505568 22365 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.591859 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:39.593710 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 110801us
I20250831 13:58:39.595595 20650 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:39.598162 20570 raft_consensus.cc:2408] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:39.599054 20571 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250831 13:58:39.571465 22361 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:39.605204 22358 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.605358 22361 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:39.606552 22361 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.597499 22372 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:39.607810 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.609476 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.610574 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:39.610806 22360 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.611006 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.611358 22371 raft_consensus.cc:695] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:39.612066 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:39.612294 22371 consensus_queue.cc:237] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.613067 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.614135 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.592664 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.616040 20493 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
W20250831 13:58:39.617143 20648 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1025) took 1341 ms (client timeout 1732 ms). Trace:
W20250831 13:58:39.617537 20648 rpcz_store.cc:269] 0831 13:58:38.275872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:38.448774 (+172902us) service_pool.cc:225] Handling call
0831 13:58:39.616973 (+1168199us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b5400372b54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:39.617094 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2219619072}
I20250831 13:58:39.617951 22066 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.619107 20642 raft_consensus.cc:2366] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4.
I20250831 13:58:39.572320 22387 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.618669 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.619899 22365 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.621704 22387 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.623961 22356 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.623430 20398 catalog_manager.cc:5582] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 3 to 4, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to ab7ade51fa134a1b814fab748f01d43d (127.18.233.132). New cstate: current_term: 4 leader_uuid: "ab7ade51fa134a1b814fab748f01d43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:58:39.624877 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.626030 20648 raft_consensus.cc:2391] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:39.626953 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.627426 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:39.628003 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:39.641135 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 758) took 1395 ms. Trace:
I20250831 13:58:39.641518 20646 rpcz_store.cc:276] 0831 13:58:38.245576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:38.996654 (+751078us) service_pool.cc:225] Handling call
0831 13:58:38.996929 (+ 275us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:39.491719 (+494790us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.640894 (+149175us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b540015b8d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:39.641073 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1130926464}
I20250831 13:58:39.643800 20576 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:39.644201 20573 raft_consensus.cc:1238] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:39.575641 22178 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:39.629011 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:39.647105 20575 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:39.628407 20648 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:39.643210 22395 consensus_queue.cc:260] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.653846 22395 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:39.654017 22342 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.656852 22278 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:39.658192 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:39.658421 22100 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.658643 22301 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.659533 22178 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:39.670320 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1020) took 1395 ms. Trace:
I20250831 13:58:39.670696 20643 rpcz_store.cc:276] 0831 13:58:38.274518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:38.754360 (+479842us) service_pool.cc:225] Handling call
0831 13:58:39.670137 (+915777us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b5400373a54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:39.670268 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1573974912}
I20250831 13:58:39.672111 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.672875 20643 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:58:39.674614 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.675364 20643 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:39.676474 20541 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.677968 20643 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250831 13:58:39.686865 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 103063us
I20250831 13:58:39.691475 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.692147 20500 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:39.696725 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.697683 20497 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:39.699498 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.704865 21588 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:39.709779 20575 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:39.710731 20617 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.713800 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:39.712929 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 204236us
W20250831 13:58:39.737902 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:39.718999 20616 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.661676 22359 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election won for term 3
I20250831 13:58:39.745321 20500 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:39.746379 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.601264 22357 consensus_queue.cc:1046] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.142s
I20250831 13:58:39.750594 20617 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.752843 20500 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20250831 13:58:39.753412 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.741205 20616 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.754961 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.717409 22188 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:39.763284 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:39.759294 20502 raft_consensus.cc:2391] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:39.765375 20502 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:39.688951 21914 consensus_queue.cc:1046] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20250831 13:58:39.767764 20500 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:39.744722 22388 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.747097 22342 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:39.774307 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.772877 20502 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:39.775732 20616 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.747545 21941 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:39.714108 22220 raft_consensus.cc:2762] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250831 13:58:39.771047 22152 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.786824 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:39.714884 22327 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.760141 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.768903 22096 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.811270 22372 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.811569 22096 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.813256 22372 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.813750 20643 raft_consensus.cc:1238] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:39.813401 20648 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:39.772192 22359 raft_consensus.cc:695] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:39.815589 22359 consensus_queue.cc:237] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.815805 21609 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.773023 22196 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.820991 20541 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:39.825323 20497 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
W20250831 13:58:39.782047 22281 consensus_meta.cc:287] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.617s user 0.009s sys 0.001s
I20250831 13:58:39.779474 22342 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.833704 22342 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:39.779726 22336 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:39.806778 22365 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:39.784159 22178 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:39.835127 20496 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:39.786299 21914 consensus_queue.cc:1230] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 13:58:39.810230 21941 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.771492 22338 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:39.806077 22182 consensus_meta.cc:287] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.548s user 0.006s sys 0.000s
I20250831 13:58:39.817330 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.847064 20577 raft_consensus.cc:2366] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6.
I20250831 13:58:39.847496 22100 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:39.774200 22157 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.744977 22229 consensus_queue.cc:237] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:39.847065 22182 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.849327 20645 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:39.850148 22338 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.850270 22182 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.794508 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:39.808005 20466 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:39.854131 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 46196.4us
I20250831 13:58:39.854952 20466 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:39.856393 20466 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:39.857226 20466 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:39.809337 21207 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250831 13:58:39.785959 22360 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:39.837234 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.842352 20569 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:39.813381 22096 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.843609 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.772871 22361 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250831 13:58:39.795900 22301 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:39.847934 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.820089 22357 consensus_queue.cc:1230] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer 212b66976f874aa682cae0fb77928786 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20250831 13:58:39.785892 21708 consensus_meta.cc:287] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.617s user 0.008s sys 0.003s
I20250831 13:58:39.851126 20646 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:39.859628 22100 raft_consensus.cc:695] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:39.851987 22378 raft_consensus.cc:3053] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:39.852118 22327 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.847909 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:39.865247 22409 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:39.870043 22375 raft_consensus.cc:3053] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:39.869938 22409 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.870388 22067 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:39.866483 20574 raft_consensus.cc:2408] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:39.865033 20497 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:39.862972 20466 leader_election.cc:400] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4.
I20250831 13:58:39.872503 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:39.868156 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.877702 20643 raft_consensus.cc:1238] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:39.878530 20466 leader_election.cc:403] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
W20250831 13:58:39.878439 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 83993us
I20250831 13:58:39.866894 22159 consensus_queue.cc:1046] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20250831 13:58:39.903517 22159 raft_consensus.cc:3053] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:39.866824 22360 raft_consensus.cc:695] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:39.903903 22159 raft_consensus.cc:738] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:39.904675 22159 consensus_queue.cc:260] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.865650 22066 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:39.905979 22159 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:39.905928 22066 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.906934 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.907575 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.904721 22360 consensus_queue.cc:237] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.903679 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 761) took 1616 ms. Trace:
I20250831 13:58:39.880424 20650 raft_consensus.cc:1273] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:39.899194 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:39.927312 20643 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:39.908121 20501 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.929323 20643 raft_consensus.cc:1273] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 13:58:39.926896 20645 rpcz_store.cc:276] 0831 13:58:38.287237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.189174 (+901937us) service_pool.cc:225] Handling call
0831 13:58:39.249178 (+ 60004us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:39.849675 (+600497us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.903441 (+ 53766us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b54002ea254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:39.903625 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1379736320}
W20250831 13:58:39.930883 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 71695.6us
I20250831 13:58:39.930943 20541 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:39.889973 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:39.819339 22152 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.931768 20466 leader_election.cc:400] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5.
W20250831 13:58:39.932096 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 52392.7us
W20250831 13:58:39.899473 21174 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:39.872208 22343 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 13:58:39.897549 21173 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:39.863174 22356 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 13:58:39.899288 22401 log.cc:927] Time spent T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.114s user 0.002s sys 0.000s
I20250831 13:58:39.837623 22238 consensus_queue.cc:1046] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20250831 13:58:39.837538 22196 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:39.841797 22281 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.870424 22375 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:39.939483 20643 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:39.881132 22209 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:39.897310 22336 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:39.866644 22100 consensus_queue.cc:237] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:39.881884 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:39.944370 20501 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:40.134920 20497 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:39.898473 22409 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:39.867383 21208 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:39.863520 22381 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:39.866493 22378 raft_consensus.cc:738] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:40.035072 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.141636 20493 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.142033 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:39.951475 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 772) took 1236 ms. Trace:
I20250831 13:58:40.142956 22409 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:40.143488 22409 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:39.948172 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.144006 22409 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:40.144376 20495 raft_consensus.cc:2391] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:40.145283 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:39.933606 22096 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4.
W20250831 13:58:40.038957 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 75532.7us
I20250831 13:58:40.141660 22378 consensus_queue.cc:260] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.147933 22378 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:39.946801 22356 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:39.938812 22343 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:40.127503 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:40.133101 22190 consensus_meta.cc:287] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.557s user 0.008s sys 0.000s
I20250831 13:58:39.881749 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:40.131845 20494 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 4. (term mismatch)
I20250831 13:58:40.134020 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 764) took 1846 ms. Trace:
I20250831 13:58:40.136499 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.170758 20494 pending_rounds.cc:85] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 3
W20250831 13:58:40.171027 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 289399us
I20250831 13:58:40.171118 20494 pending_rounds.cc:107] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.4
I20250831 13:58:40.136579 20500 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:40.142894 20646 rpcz_store.cc:276] 0831 13:58:38.715420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.642493 (+927073us) service_pool.cc:225] Handling call
0831 13:58:39.642840 (+ 347us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:39.865968 (+223128us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.951272 (+ 85304us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b540029f754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:39.951428 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":478497024}
I20250831 13:58:40.171574 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.172422 20501 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:40.172562 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:40.148684 20493 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:40.173650 20501 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250831 13:58:40.149821 20495 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:39.946275 22067 consensus_queue.cc:1035] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.182s
I20250831 13:58:39.937611 21914 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:40.148873 20617 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:40.164283 20497 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:39.935089 22342 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.188180 22286 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:40.189343 22342 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:40.189507 20650 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 13:58:39.960831 22347 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.191648 22281 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.127832 22066 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.171799 20494 raft_consensus.cc:2915] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Aborting config change with OpId 1.4: Aborted: Op aborted by new leader
I20250831 13:58:40.192153 22347 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.192505 20494 raft_consensus.cc:2887] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:40.192850 20568 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 13:58:40.193078 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52966 (request call id 414) took 3350 ms. Trace:
W20250831 13:58:40.194152 22403 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250831 13:58:40.193435 20494 rpcz_store.cc:276] 0831 13:58:36.842905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:36.847088 (+ 4183us) service_pool.cc:225] Handling call
0831 13:58:36.860263 (+ 13175us) log.cc:844] Serialized 508 byte log entry
0831 13:58:37.092549 (+232286us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b1800326320. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b9a148bf 00007fb9b9a16b61 00007fb9b9a16b19 00007fb9b9a16aa9 00007fb9b9a16a71 00007fb9b9a15d6d 0000000000379254 0000000000379099 00007fb9b99f7b1a
0831 13:58:37.099761 (+ 7212us) spinlock_profiling.cc:243] Waited 5.6 ms on lock 0x7b54000008a0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc90686f 00007fb9bc8ef502 00007fb9bc8f4911 00007fb9bc8f4899 00007fb9bc8f4829 00007fb9bc8f47f1 00007fb9bc8f39bd 0000000000379254 0000000000379099 00007fb9b9a14814
0831 13:58:40.193005 (+3093244us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":168,"mutex_wait_us":5547,"raft.queue_time_us":13855,"raft.run_cpu_time_us":3156,"raft.run_wall_time_us":93287,"spinlock_wait_cycles":16419584,"thread_start_us":162039,"threads_started":1,"wal-append.queue_time_us":297333,"wal-append.run_cpu_time_us":5109,"wal-append.run_wall_time_us":1272754}
W20250831 13:58:40.147987 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.195505 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 760) took 1908 ms. Trace:
I20250831 13:58:40.170866 20649 rpcz_store.cc:276] 0831 13:58:38.287987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.287727 (+999740us) service_pool.cc:225] Handling call
0831 13:58:39.288049 (+ 322us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:39.432015 (+143966us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:39.538278 (+106263us) log.cc:844] Serialized 14 byte log entry
0831 13:58:39.785610 (+247332us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:39.785664 (+ 54us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:39.785705 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:40.133772 (+348067us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b540015b3d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:40.133820 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:40.133949 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16828,"prepare.queue_time_us":46074,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":5422,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":14506,"spinlock_wait_cycles":479674112,"thread_start_us":333305,"threads_started":2,"wal-append.queue_time_us":327625}
I20250831 13:58:39.947090 22412 consensus_queue.cc:1046] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.189s
I20250831 13:58:40.118193 22387 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:40.127420 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 92120.7us
I20250831 13:58:40.204615 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 770) took 1559 ms. Trace:
I20250831 13:58:40.181201 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.182962 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.183763 22157 consensus_queue.cc:1046] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:40.176791 22372 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.186097 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.199883 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.269786 20643 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:40.270700 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.271471 20643 raft_consensus.cc:2391] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:40.181690 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:40.271982 20573 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:40.195937 20644 rpcz_store.cc:276] 0831 13:58:38.286912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.152191 (+865279us) service_pool.cc:225] Handling call
0831 13:58:39.153254 (+ 1063us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:39.263783 (+110529us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:39.263835 (+ 52us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:39.322467 (+ 58632us) log.cc:844] Serialized 52 byte log entry
0831 13:58:39.593222 (+270755us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:39.593265 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.953643 (+360378us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b54001ee954. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:39.953684 (+ 41us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:40.195211 (+241527us) raft_consensus.cc:1683] finished
0831 13:58:40.195231 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:40.195427 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":52072,"raft.queue_time_us":52318,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":43359,"spinlock_wait_cycles":81478016,"thread_start_us":233450,"threads_started":1,"wal-append.queue_time_us":593989}
I20250831 13:58:40.166484 22356 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.205008 20647 rpcz_store.cc:276] 0831 13:58:38.645525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.422817 (+777292us) service_pool.cc:225] Handling call
0831 13:58:39.442085 (+ 19268us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:39.442534 (+ 449us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:39.442575 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:39.443656 (+ 1081us) log.cc:844] Serialized 26 byte log entry
0831 13:58:39.561825 (+118169us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:39.561877 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:39.561956 (+ 79us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:39.898761 (+336805us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:40.204358 (+305597us) raft_consensus.cc:1683] finished
0831 13:58:40.204382 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:40.204559 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23221,"raft.queue_time_us":94,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":6349,"spinlock_wait_cycles":55933184,"thread_start_us":79231,"threads_started":1,"wal-append.queue_time_us":327271}
I20250831 13:58:40.181501 22190 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5.
I20250831 13:58:40.238528 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:40.239289 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.278934 20572 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:40.195452 22152 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.243055 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:40.246788 22380 consensus_meta.cc:287] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.085s user 0.006s sys 0.006s
I20250831 13:58:40.169723 22343 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.252534 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:40.253788 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:40.294173 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.296802 20646 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.291608 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:40.180305 21914 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:40.177326 22336 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:40.126330 22221 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20250831 13:58:40.134178 21708 consensus_queue.cc:1046] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
W20250831 13:58:40.238080 20387 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 90a0a56228d04660b3cb351485be6593 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) due to error UNKNOWN_ERROR; will retry: Aborted: Op aborted by new leader
I20250831 13:58:40.129065 22388 consensus_queue.cc:1035] T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20250831 13:58:40.277669 20644 raft_consensus.cc:1273] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:40.313586 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:40.284368 20575 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:40.266562 22405 log.cc:927] Time spent T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.412s user 0.000s sys 0.001s
I20250831 13:58:40.298573 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.297533 22374 raft_consensus.cc:3053] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:40.306949 20647 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250831 13:58:40.293138 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.317484 20576 raft_consensus.cc:2408] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:40.307838 20646 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:40.311323 20466 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:40.314603 20616 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:40.288899 22395 consensus_meta.cc:287] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.635s user 0.011s sys 0.000s
I20250831 13:58:40.293644 22409 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:40.316829 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.289716 21609 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.328622 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.321213 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:40.335162 20651 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:40.326180 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:40.318496 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:40.325522 20387 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 123623us
I20250831 13:58:40.315997 20572 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:40.337210 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:40.283643 22372 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.337893 20572 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:40.338981 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.339905 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.340390 20502 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:40.340840 20575 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:40.289386 21941 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.288843 22178 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.343117 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.283452 22387 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.347905 20647 raft_consensus.cc:2408] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:40.343653 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.350144 20646 raft_consensus.cc:1273] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:40.353581 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:40.354156 20644 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 6
W20250831 13:58:40.355739 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 44456.4us
I20250831 13:58:40.356066 20642 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:40.357468 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:40.357265 20466 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:40.310289 22406 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.359892 20617 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:40.360168 20648 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:40.342540 22286 raft_consensus.cc:695] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:40.360883 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.306324 21914 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:40.378634 22395 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:40.381966 22395 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.384351 22395 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.387462 22409 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:40.363240 22394 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:40.388324 22394 raft_consensus.cc:695] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:40.389194 22394 consensus_queue.cc:237] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:40.391678 22419 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.526s user 0.001s sys 0.000s
W20250831 13:58:40.395845 22365 consensus_meta.cc:287] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.561s user 0.008s sys 0.000s
I20250831 13:58:40.396231 22365 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:40.337003 22157 raft_consensus.cc:3053] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:40.363509 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:40.396835 22157 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:40.334712 22374 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:40.319208 22412 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:40.365284 20647 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 4 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 13:58:40.359709 22178 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.366382 22286 consensus_queue.cc:237] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.371788 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:40.360064 21941 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:40.336722 22066 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:40.271005 22375 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:40.376945 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:40.430619 20543 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.526s (SENT) [suppressed 2 similar messages]
I20250831 13:58:40.430836 20649 raft_consensus.cc:1273] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:40.395648 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.376128 20617 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:40.292464 22380 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.402349 20570 raft_consensus.cc:3053] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:40.414214 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.363739 22196 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:40.405045 21941 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:40.397656 22157 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.406865 22229 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:40.433947 20570 raft_consensus.cc:738] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:58:40.436666 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 123911us
W20250831 13:58:40.436971 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 71052.6us
I20250831 13:58:40.438846 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.464615 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.423678 22178 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:40.437645 21941 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.433099 22329 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:40.494791 20573 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:40.495864 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.501575 20647 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:40.498977 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:40.499477 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.501103 20496 raft_consensus.cc:2391] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:40.502875 20466 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:40.504349 22381 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:58:40.504801 22381 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:40.505221 22381 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:40.365752 22406 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.514904 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:40.437561 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 112969us
I20250831 13:58:40.411191 22412 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.485746 22431 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 13:58:40.513147 21173 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.487924 21609 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.532080 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.533140 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.437101 22066 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.534166 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.512072 20570 consensus_queue.cc:260] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.535131 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.535158 20570 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.535527 22066 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.535702 20500 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:40.492234 22229 raft_consensus.cc:695] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:40.539505 22229 consensus_queue.cc:237] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.546279 20500 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:40.547724 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.549751 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:40.551348 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.556036 20647 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250831 13:58:40.512037 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 168989us
I20250831 13:58:40.561614 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.562006 20570 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250831 13:58:40.561527 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.562542 20570 pending_rounds.cc:85] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 3
I20250831 13:58:40.563287 20570 pending_rounds.cc:107] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250831 13:58:40.563737 20570 raft_consensus.cc:2887] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:40.564003 20572 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:58:40.565591 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.567375 21992 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:40.568733 22412 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.555195 21609 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:40.570917 21609 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:40.571265 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.571897 20577 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:40.573088 20577 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20250831 13:58:40.566560 20572 raft_consensus.cc:2391] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:40.577950 20644 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:40.579448 22375 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:40.579550 22429 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.579864 20502 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:40.580128 22429 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:40.580714 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.599282 20502 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
W20250831 13:58:40.600116 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.600687 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:40.601693 22429 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:40.601907 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:40.602365 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.438983 22374 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.602777 21208 raft_consensus.cc:2408] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 13:58:40.589126 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.603430 22374 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:40.603951 22432 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20250831 13:58:40.604485 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.605018 20502 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.605332 20568 raft_consensus.cc:1238] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:40.605549 20574 raft_consensus.cc:1238] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:40.581043 22078 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.606355 20568 raft_consensus.cc:1238] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:40.606216 22078 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:40.606856 20574 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
W20250831 13:58:40.584074 22418 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.718s user 0.000s sys 0.002s
I20250831 13:58:40.607251 20574 pending_rounds.cc:85] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 2
W20250831 13:58:40.590399 22408 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.721s user 0.007s sys 0.002s
I20250831 13:58:40.607692 20574 pending_rounds.cc:107] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 13:58:40.608120 20574 raft_consensus.cc:2887] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:40.597805 21941 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.583009 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:40.609560 20577 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:40.609855 21941 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.610459 22342 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:40.610852 22342 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:40.611174 22342 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:40.585618 22347 raft_consensus.cc:3053] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:40.612680 22347 raft_consensus.cc:738] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:40.583312 22357 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.613324 22357 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:40.613518 22347 consensus_queue.cc:260] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.614404 22347 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:39.822839 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:58:40.580250 20644 raft_consensus.cc:1273] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:40.617678 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.618112 22380 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.618891 22436 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:40.621096 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1051) took 1163 ms. Trace:
I20250831 13:58:40.605015 22372 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:40.621551 20650 rpcz_store.cc:276] 0831 13:58:39.457346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.932780 (+475434us) service_pool.cc:225] Handling call
0831 13:58:39.940436 (+ 7656us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:40.190011 (+249575us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:40.620786 (+430775us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b540015f254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:40.621013 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":572256128}
I20250831 13:58:40.526476 22336 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:40.602977 22387 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.562553 22431 raft_consensus.cc:695] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:40.627266 22372 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:40.627426 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 789) took 1123 ms. Trace:
I20250831 13:58:40.627965 20651 rpcz_store.cc:276] 0831 13:58:39.503739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.583117 (+ 79378us) service_pool.cc:225] Handling call
0831 13:58:40.627216 (+1044099us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b540029f254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:40.627360 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1652154112}
I20250831 13:58:40.631596 20495 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:40.631806 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.632788 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.633268 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:40.634790 20651 raft_consensus.cc:1238] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:40.634505 20495 raft_consensus.cc:1238] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:40.635540 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1149) took 1025 ms. Trace:
I20250831 13:58:40.564445 22196 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.051s
I20250831 13:58:40.636336 20495 raft_consensus.cc:1238] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:40.635926 20569 rpcz_store.cc:276] 0831 13:58:39.610416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.634444 (+ 24028us) service_pool.cc:225] Handling call
0831 13:58:39.634820 (+ 376us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:39.863372 (+228552us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:40.635338 (+771966us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54001a6dd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:40.635483 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":476768000}
W20250831 13:58:40.639037 22426 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.333s user 0.003s sys 0.000s
I20250831 13:58:40.640465 22410 consensus_queue.cc:1046] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20250831 13:58:40.641439 22420 log.cc:927] Time spent T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.773s user 0.000s sys 0.004s
I20250831 13:58:40.641903 22410 consensus_queue.cc:1046] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:40.643373 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1049) took 1186 ms. Trace:
I20250831 13:58:40.643539 22387 raft_consensus.cc:3053] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:40.643966 22387 raft_consensus.cc:738] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:40.644317 20647 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250831 13:58:40.644807 22387 consensus_queue.cc:260] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.628261 22431 consensus_queue.cc:237] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.647651 22387 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:40.647934 22157 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:40.643838 20645 rpcz_store.cc:276] 0831 13:58:39.456713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.931447 (+474734us) service_pool.cc:225] Handling call
0831 13:58:39.943851 (+ 12404us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:39.944148 (+ 297us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:39.944166 (+ 18us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:39.950440 (+ 6274us) log.cc:844] Serialized 26 byte log entry
0831 13:58:40.109620 (+159180us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:40.129019 (+ 19399us) log.cc:844] Serialized 14 byte log entry
0831 13:58:40.132454 (+ 3435us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:40.132517 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:40.638781 (+506264us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x7b20001b1878. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc5c8bf1 00007fb9bc5eaa29 00007fb9bc5b5973 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209 00007fb9bc5be1d1 00007fb9bc5bd4cd 0000000000379254
0831 13:58:40.643104 (+ 4323us) raft_consensus.cc:1683] finished
0831 13:58:40.643195 (+ 91us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:40.643317 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3142,"prepare.queue_time_us":19355,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":3309,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":19219,"spinlock_wait_cycles":18295040,"thread_start_us":158895,"threads_started":1,"wal-append.queue_time_us":183494}
W20250831 13:58:40.591552 21588 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20250831 13:58:40.651808 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:40.652338 20498 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:40.656924 22416 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.791s user 0.002s sys 0.000s
I20250831 13:58:40.657071 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46464 (request call id 45) took 1132 ms. Trace:
I20250831 13:58:40.657418 20648 rpcz_store.cc:276] 0831 13:58:39.524695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.275538 (+750843us) service_pool.cc:225] Handling call
0831 13:58:40.275890 (+ 352us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:40.362077 (+ 86187us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:40.656830 (+294753us) spinlock_profiling.cc:243] Waited 84.1 ms on lock 0x7b54003a11d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:40.657010 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":193370368}
I20250831 13:58:40.661782 20499 raft_consensus.cc:1273] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:40.663024 22301 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:40.663514 22301 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.595243 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:40.668336 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.607164 20500 raft_consensus.cc:2391] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:40.669785 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 771) took 1955 ms. Trace:
I20250831 13:58:40.670270 20642 rpcz_store.cc:276] 0831 13:58:38.714668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.633799 (+919131us) service_pool.cc:225] Handling call
0831 13:58:39.634101 (+ 302us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:40.356432 (+722331us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:40.669504 (+313072us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b540030f7d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:40.669721 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1659895680}
I20250831 13:58:40.675158 22196 raft_consensus.cc:3053] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:40.675590 22196 raft_consensus.cc:738] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:40.676520 22196 consensus_queue.cc:260] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.677948 22196 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:40.678895 20500 raft_consensus.cc:1273] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:40.681640 20500 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:40.685833 20500 raft_consensus.cc:1273] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20250831 13:58:40.687225 20617 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:40.688627 20495 raft_consensus.cc:1273] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250831 13:58:40.732718 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 45631.6us
I20250831 13:58:40.689220 22078 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:40.627987 22372 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.691293 22406 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:40.692440 22159 consensus_meta.cc:287] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.786s user 0.012s sys 0.000s
I20250831 13:58:40.693249 22357 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:58:40.698827 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 268234us
I20250831 13:58:40.712059 20499 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:40.719098 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.721675 20398 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54880 (request call id 38) took 3097 ms. Trace:
I20250831 13:58:40.722436 20496 raft_consensus.cc:2408] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:40.723518 20647 raft_consensus.cc:1273] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:40.831391 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:40.832610 20647 raft_consensus.cc:2391] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:40.702499 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:40.727398 20466 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:40.734845 20500 pending_rounds.cc:85] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 2
I20250831 13:58:40.743301 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:40.842872 20649 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:40.743798 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:40.843756 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.830063 20398 rpcz_store.cc:276] 0831 13:58:37.624612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:37.624774 (+ 162us) service_pool.cc:225] Handling call
0831 13:58:40.721600 (+3096826us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:39.779150 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:39.831446 (+ 52296us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:39.831487 (+ 41us) write_op.cc:435] Acquired schema lock
0831 13:58:39.831503 (+ 16us) tablet.cc:660] Decoding operations
0831 13:58:39.831909 (+ 406us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:39.831965 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:39.831998 (+ 33us) tablet.cc:683] Acquiring locks for 2 operations
0831 13:58:39.832297 (+ 299us) tablet.cc:699] Row locks acquired
0831 13:58:39.832312 (+ 15us) write_op.cc:260] PREPARE: finished
0831 13:58:39.832449 (+ 137us) write_op.cc:270] Start()
0831 13:58:39.832560 (+ 111us) write_op.cc:276] Timestamp: P: 1756648719832406 usec, L: 0
0831 13:58:39.832600 (+ 40us) op_driver.cc:348] REPLICATION: starting
0831 13:58:39.833267 (+ 667us) log.cc:844] Serialized 924 byte log entry
0831 13:58:40.275258 (+441991us) op_driver.cc:464] REPLICATION: finished
0831 13:58:40.630721 (+355463us) write_op.cc:301] APPLY: starting
0831 13:58:40.630822 (+ 101us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:40.630966 (+ 144us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:40.631118 (+ 152us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:40.631411 (+ 293us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:40.631555 (+ 144us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0831 13:58:40.631594 (+ 39us) write_op.cc:312] APPLY: finished
0831 13:58:40.631763 (+ 169us) log.cc:844] Serialized 28 byte log entry
0831 13:58:40.631930 (+ 167us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:40.632007 (+ 77us) write_op.cc:454] Released schema lock
0831 13:58:40.632041 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:40.632406 (+ 365us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:40.632433 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":526489,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":355170,"apply.run_cpu_time_us":1709,"apply.run_wall_time_us":1709,"num_ops":2,"prepare.queue_time_us":123040,"prepare.run_cpu_time_us":2873,"prepare.run_wall_time_us":73351,"replication_time_us":375364,"thread_start_us":295640,"threads_started":2}]]}
I20250831 13:58:40.753371 20502 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:40.789594 20576 raft_consensus.cc:2408] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:40.788661 20650 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:40.744392 20397 catalog_manager.cc:5582] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:40.849582 20397 catalog_manager.cc:5582] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 4 to 5, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 5 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
W20250831 13:58:40.778097 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 288972us
I20250831 13:58:40.737416 22338 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:40.711730 22365 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.850641 20502 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:40.754701 22369 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.852491 20397 catalog_manager.cc:5582] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
W20250831 13:58:40.861779 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 134513us
W20250831 13:58:40.862246 20396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59926 (request call id 133) took 13162 ms (client timeout 15000 ms). Trace:
W20250831 13:58:40.862361 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 159934us
W20250831 13:58:40.862583 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.851037 20500 pending_rounds.cc:107] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20250831 13:58:40.866268 20500 raft_consensus.cc:2887] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:40.866498 20494 raft_consensus.cc:1273] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250831 13:58:40.867442 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:40.867834 20496 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 13:58:40.868970 20467 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:40.869817 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.870734 20648 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 4: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:40.870808 20502 raft_consensus.cc:1273] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250831 13:58:40.864640 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.865387 20397 catalog_manager.cc:5582] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:40.793912 22406 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.872849 20645 raft_consensus.cc:2391] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:40.872977 20499 raft_consensus.cc:1273] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250831 13:58:40.884950 20651 raft_consensus.cc:2953] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Committing config change with OpId 5.5: config changed from index 2 to 5, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:40.747479 22336 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20250831 13:58:40.743620 22356 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.891546 20570 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:40.872594 20397 catalog_manager.cc:5582] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 reported cstate change: term changed from 2 to 3, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:40.907684 20397 catalog_manager.cc:5582] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:40.909395 20397 catalog_manager.cc:5582] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 reported cstate change: term changed from 3 to 5, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 5 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:40.859391 22430 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:40.913885 20500 raft_consensus.cc:1273] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20250831 13:58:40.844354 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 51333.7us
W20250831 13:58:40.912760 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 43853us
W20250831 13:58:40.916122 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:40.921185 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.886853 22410 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:40.874193 22357 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:40.922887 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:40.938225 20497 raft_consensus.cc:1238] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20250831 13:58:40.910862 22394 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.012s sys 0.012s
I20250831 13:58:40.915848 20397 catalog_manager.cc:5582] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 2 to 3, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:40.798198 22358 consensus_queue.cc:1035] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20250831 13:58:40.931936 20649 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:40.870535 22429 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.902483 22406 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:40.888085 22365 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:40.890867 22182 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20250831 13:58:40.884436 22369 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:41.011122 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 88366us
W20250831 13:58:41.011402 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 56146.1us
I20250831 13:58:40.903684 22159 consensus_queue.cc:1035] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
I20250831 13:58:40.915814 22152 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:40.951068 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:40.955622 22078 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:40.955760 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.997072 20397 catalog_manager.cc:5582] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 4, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 4 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
W20250831 13:58:40.940371 22394 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s user 0.012s sys 0.012s
W20250831 13:58:40.939672 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.936367 22188 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20250831 13:58:41.031741 22406 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.939810 22380 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.086s
I20250831 13:58:40.935297 22430 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.032366 21205 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 13:58:41.003602 20498 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:41.033648 22430 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:41.033821 20493 raft_consensus.cc:1273] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:40.937201 22360 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20250831 13:58:41.006816 20466 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:41.009095 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:41.007699 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 172875us
I20250831 13:58:40.933590 22371 consensus_queue.cc:1035] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20250831 13:58:40.944046 22357 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:41.013726 20467 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:40.947986 21941 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.945462 22372 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:40.944597 22356 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:40.955855 22443 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.918494 22378 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
W20250831 13:58:40.865077 20396 rpcz_store.cc:269] 0831 13:58:27.699528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:27.766930 (+ 67402us) service_pool.cc:225] Handling call
0831 13:58:40.862188 (+13095258us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:37.559747 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:37.560341 (+ 594us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:37.560373 (+ 32us) write_op.cc:435] Acquired schema lock
0831 13:58:37.560391 (+ 18us) tablet.cc:660] Decoding operations
0831 13:58:37.561022 (+ 631us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:37.561089 (+ 67us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:37.561112 (+ 23us) tablet.cc:683] Acquiring locks for 5 operations
0831 13:58:37.561642 (+ 530us) tablet.cc:699] Row locks acquired
0831 13:58:37.561660 (+ 18us) write_op.cc:260] PREPARE: finished
0831 13:58:37.561796 (+ 136us) write_op.cc:270] Start()
0831 13:58:37.561905 (+ 109us) write_op.cc:276] Timestamp: P: 1756648717561747 usec, L: 0
0831 13:58:37.561921 (+ 16us) op_driver.cc:348] REPLICATION: starting
0831 13:58:37.563124 (+ 1203us) log.cc:844] Serialized 2108 byte log entry
0831 13:58:38.148792 (+585668us) op_driver.cc:464] REPLICATION: finished
0831 13:58:38.628983 (+480191us) write_op.cc:301] APPLY: starting
0831 13:58:38.632750 (+ 3767us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:38.658160 (+ 25410us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:38.725089 (+ 66929us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:38.888518 (+163429us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:38.888714 (+ 196us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0831 13:58:38.888755 (+ 41us) write_op.cc:312] APPLY: finished
0831 13:58:38.888959 (+ 204us) log.cc:844] Serialized 46 byte log entry
0831 13:58:38.964104 (+ 75145us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:38.964249 (+ 145us) write_op.cc:454] Released schema lock
0831 13:58:38.964363 (+ 114us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:39.041968 (+ 77605us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:39.042020 (+ 52us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":293301,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":458178,"apply.run_cpu_time_us":10303,"apply.run_wall_time_us":427618,"num_ops":5,"prepare.queue_time_us":224578,"prepare.run_cpu_time_us":3227,"prepare.run_wall_time_us":35533,"raft.queue_time_us":331561,"raft.run_cpu_time_us":61086,"raft.run_wall_time_us":340538,"replication_time_us":540557,"thread_start_us":551622,"threads_started":4,"wal-append.queue_time_us":94570}]]}
I20250831 13:58:40.940285 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:58:41.017338 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 66379.8us
I20250831 13:58:41.017668 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:40.948117 21992 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:40.955118 22206 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:41.023170 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:40.950984 22402 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:41.023777 20397 catalog_manager.cc:5582] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 reported cstate change: term changed from 1 to 3, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
W20250831 13:58:41.078874 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 61378.8us
I20250831 13:58:41.079079 21992 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.080933 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:41.081045 20543 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.081286 21992 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.016254 22157 consensus_queue.cc:1046] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20250831 13:58:41.027989 22365 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:40.973986 22229 consensus_queue.cc:1035] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20250831 13:58:40.948848 22327 consensus_queue.cc:1035] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.073s
W20250831 13:58:41.011523 20646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.034s sys 0.014s
W20250831 13:58:41.084129 20646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s user 0.034s sys 0.015s
W20250831 13:58:41.038556 20498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 760) took 1292 ms (client timeout 1526 ms). Trace:
W20250831 13:58:41.084578 20498 rpcz_store.cc:269] 0831 13:58:39.746357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:39.746478 (+ 121us) service_pool.cc:225] Handling call
0831 13:58:40.651680 (+905202us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:41.038497 (+386817us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12244864}
W20250831 13:58:41.022269 22381 consensus_meta.cc:287] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.517s user 0.009s sys 0.002s
I20250831 13:58:41.060240 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:41.062546 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 48901.8us
I20250831 13:58:40.951403 22367 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.178s
W20250831 13:58:41.037456 21914 consensus_meta.cc:287] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.601s user 0.008s sys 0.001s
I20250831 13:58:41.067413 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:41.036484 22078 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.034931 22067 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:41.085968 20543 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:41.043720 22388 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:41.017127 22347 consensus_queue.cc:1046] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20250831 13:58:41.078899 22206 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.083262 20541 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:41.069911 22335 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:41.096948 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:41.106207 20649 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
W20250831 13:58:41.082829 22343 consensus_meta.cc:287] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.767s user 0.010s sys 0.000s
I20250831 13:58:41.098335 20502 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:40.996382 22429 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:41.098953 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:41.095345 22329 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:41.044529 22406 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:41.079243 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:41.107286 20466 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:41.067024 21941 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.108691 21208 raft_consensus.cc:2391] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
I20250831 13:58:41.091975 22238 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:41.132764 20645 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:58:41.135236 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:41.108896 22375 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 13:58:41.113945 22447 log.cc:927] Time spent T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20250831 13:58:41.108361 22374 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:41.140558 20616 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.149041 20647 raft_consensus.cc:1273] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:41.150352 20616 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.151656 20616 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.099215 22281 consensus_queue.cc:1035] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20250831 13:58:41.144028 20648 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 13:58:41.161321 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:41.139024 22375 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:41.161715 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 68298.8us
I20250831 13:58:41.075579 22443 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.139071 22369 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:41.153900 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:41.155496 20616 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.096007 22381 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:41.108536 22067 raft_consensus.cc:695] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:41.165452 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 779) took 1217 ms. Trace:
I20250831 13:58:41.100958 21914 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.248474 22371 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:41.249363 21914 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.250113 21914 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:41.166791 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:41.252945 20571 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250831 13:58:41.107416 22388 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:41.154000 22441 log.cc:927] Time spent T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250831 13:58:41.114238 22343 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.255033 22388 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.255232 22343 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.255843 20501 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:41.192518 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:41.256883 20493 raft_consensus.cc:2408] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:41.258318 20493 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:41.160751 22335 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:41.163666 22182 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:58:41.228387 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 119031us
W20250831 13:58:41.246742 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 61169.6us
I20250831 13:58:41.259815 20543 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.260383 20616 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.260857 20543 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:41.261358 22358 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:41.261924 22358 raft_consensus.cc:695] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:41.261812 22182 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:41.168568 22372 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:41.163633 22406 raft_consensus.cc:695] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:41.259198 22335 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:41.163993 22375 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:41.194305 20647 raft_consensus.cc:1238] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:41.274928 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:41.275753 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1198) took 1076 ms. Trace:
I20250831 13:58:41.256588 20497 rpcz_store.cc:276] 0831 13:58:39.948061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.202810 (+254749us) service_pool.cc:225] Handling call
0831 13:58:40.203435 (+ 625us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:40.938667 (+735232us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:41.165230 (+226563us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b54001293d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:41.165402 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1689324800}
I20250831 13:58:41.167312 22374 raft_consensus.cc:695] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:41.250669 21914 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.148903 22238 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20250831 13:58:41.180190 22229 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:41.181293 22157 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:41.193403 22342 consensus_meta.cc:287] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.582s user 0.011s sys 0.000s
W20250831 13:58:41.272711 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 107843us
I20250831 13:58:41.276276 20575 rpcz_store.cc:276] 0831 13:58:40.199784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.341640 (+141856us) service_pool.cc:225] Handling call
0831 13:58:40.341989 (+ 349us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:40.343305 (+ 1316us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:40.343348 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:40.355765 (+ 12417us) log.cc:844] Serialized 26 byte log entry
0831 13:58:40.882955 (+527190us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:40.883012 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:40.883109 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:41.275486 (+392377us) raft_consensus.cc:1683] finished
0831 13:58:41.275515 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:41.275676 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":887,"raft.queue_time_us":4216,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":11831,"thread_start_us":473491,"threads_started":1,"wal-append.queue_time_us":622988}
I20250831 13:58:41.287277 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:41.289746 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:41.294759 22327 consensus_queue.cc:1046] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20250831 13:58:41.318478 20467 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:41.316666 20576 raft_consensus.cc:1273] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:41.270227 22360 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20250831 13:58:41.319187 20647 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:41.295217 22066 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:41.320032 22327 consensus_queue.cc:1230] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 13:58:41.317277 20573 raft_consensus.cc:1273] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:41.194289 22378 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
I20250831 13:58:41.336933 20577 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 13:58:41.281944 22335 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:41.339555 21588 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:41.319000 22342 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.330206 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:41.332551 20467 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:41.332566 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:41.334034 20466 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:41.320036 22455 log.cc:927] Time spent T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
I20250831 13:58:41.337033 20573 raft_consensus.cc:1273] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250831 13:58:41.353341 20573 pending_rounds.cc:85] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 3
I20250831 13:58:41.353566 20573 pending_rounds.cc:107] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250831 13:58:41.353777 20573 raft_consensus.cc:2887] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:41.340564 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:41.325011 20647 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20250831 13:58:41.323619 22387 consensus_meta.cc:287] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.674s user 0.011s sys 0.000s
I20250831 13:58:41.263236 22358 consensus_queue.cc:237] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.347905 20568 raft_consensus.cc:1273] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250831 13:58:41.350318 20467 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:41.362843 20568 pending_rounds.cc:85] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 2
I20250831 13:58:41.363214 20568 pending_rounds.cc:107] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 13:58:41.363487 20568 raft_consensus.cc:2887] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:41.353695 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:41.294128 22347 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:41.321262 22229 raft_consensus.cc:695] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:41.374642 20577 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250831 13:58:41.375093 20577 pending_rounds.cc:85] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 2
I20250831 13:58:41.375388 20577 pending_rounds.cc:107] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 13:58:41.375111 20467 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:41.375866 20577 raft_consensus.cc:2887] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:41.364980 20574 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:41.368180 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:41.366540 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:41.335384 22343 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:41.335031 22281 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:41.322074 22157 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.388979 20573 raft_consensus.cc:1273] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20250831 13:58:41.357024 22459 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:41.403180 20573 pending_rounds.cc:85] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 2
I20250831 13:58:41.411188 20573 pending_rounds.cc:107] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250831 13:58:41.411743 20573 raft_consensus.cc:2887] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:41.360288 22394 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:41.394100 22410 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:41.394937 22387 consensus_queue.cc:1046] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:41.395104 22380 consensus_queue.cc:1046] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:41.402093 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:41.402326 21588 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:41.392768 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:41.375157 22078 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.366315 22336 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:41.430768 21588 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.375267 22429 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.415004 22459 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:41.427268 22301 consensus_meta.cc:287] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.763s user 0.006s sys 0.005s
I20250831 13:58:41.444196 21588 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.366700 22357 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20250831 13:58:41.379418 21941 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.386395 22206 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:41.368065 22066 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.409755 22468 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:41.453946 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:41.472960 20568 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:41.384573 22458 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:41.348452 22067 consensus_queue.cc:237] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.457216 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:41.379173 22347 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:41.487394 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:41.448724 22459 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:41.450907 22286 consensus_queue.cc:1035] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.005s
I20250831 13:58:41.385419 22374 consensus_queue.cc:237] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.366333 22406 consensus_queue.cc:237] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.448180 22338 consensus_queue.cc:1035] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20250831 13:58:41.443162 22281 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20250831 13:58:41.519977 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:41.463210 22453 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.305s user 0.000s sys 0.000s
I20250831 13:58:41.536072 20642 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:41.538046 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:41.504652 22451 log.cc:927] Time spent T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.339s user 0.001s sys 0.001s
I20250831 13:58:41.565562 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:41.519313 22449 log.cc:927] Time spent T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.245s user 0.001s sys 0.000s
I20250831 13:58:41.466943 22182 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:41.443374 22336 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:41.519655 22466 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.239s user 0.001s sys 0.000s
W20250831 13:58:41.517293 22463 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
W20250831 13:58:41.520706 22448 log.cc:927] Time spent T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.328s user 0.000s sys 0.001s
W20250831 13:58:41.520004 22444 log.cc:927] Time spent T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.232s user 0.001s sys 0.000s
I20250831 13:58:41.451558 22301 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.457747 22360 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
I20250831 13:58:41.582296 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:41.579989 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:41.585282 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 143498us
I20250831 13:58:41.620294 20616 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.519562 22157 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.530508 22443 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:41.570135 22196 consensus_meta.cc:287] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.892s user 0.008s sys 0.006s
I20250831 13:58:41.506248 22372 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20250831 13:58:41.619326 20642 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:41.627311 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:41.627919 20642 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:41.577804 22473 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:41.617887 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 357081us
I20250831 13:58:41.616365 21914 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.520494 22229 consensus_queue.cc:237] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.624358 22359 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:41.586885 22182 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.932816 20543 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.933106 22359 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.934098 20543 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.934392 22182 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:41.565135 22468 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.935079 22359 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:41.935156 22327 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:41.633358 22381 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:41.935840 22327 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:41.720643 22372 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20250831 13:58:41.936335 22468 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.635004 22473 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.938774 22473 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.664579 22342 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:41.740852 22476 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20250831 13:58:41.939608 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:41.741883 22475 log.cc:927] Time spent T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.161s user 0.000s sys 0.000s
W20250831 13:58:41.742070 22460 log.cc:927] Time spent T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
I20250831 13:58:41.940184 20499 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:41.635905 22365 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
W20250831 13:58:41.751507 22464 log.cc:927] Time spent T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
I20250831 13:58:41.752724 22336 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:41.752904 22450 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:41.942790 22450 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:41.945009 22450 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.944926 20543 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.766108 22066 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.635149 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:41.745473 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1079) took 1020 ms. Trace:
W20250831 13:58:41.742333 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 118755us
I20250831 13:58:41.951151 20644 rpcz_store.cc:276] 0831 13:58:40.725212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.745581 (+ 20369us) service_pool.cc:225] Handling call
0831 13:58:40.745946 (+ 365us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:40.793243 (+ 47297us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:41.097588 (+304345us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b18004726c0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a11805 00007fb9c035b327 00007fb9bc980957 00007fb9bc99006f 00007fb9bc98ffe8
0831 13:58:41.351748 (+254160us) log.cc:844] Serialized 14 byte log entry
0831 13:58:41.532213 (+180465us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:41.536614 (+ 4401us) log.cc:844] Serialized 26 byte log entry
0831 13:58:41.536819 (+ 205us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:41.536922 (+ 103us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:41.537005 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:41.745169 (+208164us) raft_consensus.cc:1683] finished
0831 13:58:41.745272 (+ 103us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:41.745414 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59117,"prepare.queue_time_us":602866,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":6442,"raft.queue_time_us":158234,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":19964,"spinlock_wait_cycles":14734720,"thread_start_us":428659,"threads_started":2,"wal-append.queue_time_us":268847}
I20250831 13:58:41.952797 20616 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.631053 20647 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:41.957208 20647 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250831 13:58:41.738212 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1056) took 1782 ms. Trace:
I20250831 13:58:41.960603 20646 rpcz_store.cc:276] 0831 13:58:39.956151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.351528 (+395377us) service_pool.cc:225] Handling call
0831 13:58:40.351723 (+ 195us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:40.351988 (+ 265us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:40.352014 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:40.352401 (+ 387us) log.cc:844] Serialized 52 byte log entry
0831 13:58:41.092470 (+740069us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:41.092509 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:41.092571 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:41.737926 (+645355us) raft_consensus.cc:1683] finished
0831 13:58:41.737958 (+ 32us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:41.738135 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2894,"raft.queue_time_us":79,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":5974,"thread_start_us":681520,"threads_started":1,"wal-append.queue_time_us":769856}
I20250831 13:58:41.782493 22375 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.964478 22375 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.785193 22335 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.786020 22459 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.804857 22347 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:41.861764 20541 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:41.965768 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 104049us
I20250831 13:58:41.966385 20541 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:41.966620 22459 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:41.880930 22446 log.cc:927] Time spent T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.600s user 0.003s sys 0.000s
I20250831 13:58:41.884999 22374 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:41.968858 22374 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.969327 22473 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:41.969573 22066 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:41.970176 22066 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:41.970599 22066 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:41.970866 22374 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.971644 22374 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:41.972087 22374 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:41.973960 22374 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.974022 22335 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.754279 22301 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:41.969748 22473 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:41.974817 22473 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:41.975847 22483 log.cc:927] Time spent T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.239s user 0.002s sys 0.001s
I20250831 13:58:41.946386 22472 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:41.980434 20616 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.980840 22336 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:41.987862 22066 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:41.987394 22492 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.232s user 0.000s sys 0.002s
I20250831 13:58:41.990608 22066 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:41.992672 22301 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:41.993748 20616 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:41.950279 20543 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:41.997531 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:42.001739 22347 consensus_queue.cc:1046] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:41.998359 20646 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:42.000973 22375 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:58:42.003036 22375 raft_consensus.cc:3053] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:42.003124 22347 raft_consensus.cc:3053] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:42.003607 22347 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:42.004650 22347 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.005090 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:42.005707 22347 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:42.005920 20646 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:42.007328 22374 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:42.003530 22375 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:42.008678 22375 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.009824 22375 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:58:42.008701 22488 log.cc:927] Time spent T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.255s user 0.000s sys 0.002s
I20250831 13:58:42.023224 22374 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 13:58:42.032720 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 82519.9us
I20250831 13:58:42.033743 20543 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:42.001448 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:42.035693 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:42.037492 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:42.039599 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 45948.1us
I20250831 13:58:42.040870 22337 consensus_queue.cc:1035] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:42.047664 22338 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:42.048239 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:42.049392 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:42.056383 20649 raft_consensus.cc:2366] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:42.057096 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:42.057477 20541 leader_election.cc:400] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:42.057660 20649 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 5.
I20250831 13:58:42.057895 20541 leader_election.cc:403] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20250831 13:58:42.058650 22375 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:42.059603 21174 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:42.060418 22450 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:42.060848 22450 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:42.061230 22450 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:42.049949 22336 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:42.050632 20616 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:42.053915 22159 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:42.063701 20616 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:42.064538 22336 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:42.064906 22336 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:42.065223 22336 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:42.058568 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:42.072420 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:42.072599 20649 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:42.086414 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:42.087399 20574 raft_consensus.cc:2408] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 13:58:42.088419 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:42.074087 22365 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:42.111475 20541 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:42.097401 20646 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:42.098469 20574 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:42.113917 20541 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:42.118006 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:42.120434 20616 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:42.122889 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:42.124437 20574 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:42.130992 20616 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:42.119930 22374 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:42.139766 22374 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:42.139794 22229 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:42.140385 22229 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:42.140808 22229 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:42.140197 22374 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:42.145337 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1230) took 1073 ms. Trace:
I20250831 13:58:42.149905 20500 rpcz_store.cc:276] 0831 13:58:41.071870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.081759 (+ 9889us) service_pool.cc:225] Handling call
0831 13:58:41.082087 (+ 328us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:41.091358 (+ 9271us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:41.091393 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:41.091800 (+ 407us) log.cc:844] Serialized 26 byte log entry
0831 13:58:41.134014 (+ 42214us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:41.134060 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:41.134132 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:42.145071 (+1010939us) raft_consensus.cc:1683] finished
0831 13:58:42.145098 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:42.145267 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8185,"raft.queue_time_us":9240,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":23234,"threads_started":1,"wal-append.queue_time_us":66166}
I20250831 13:58:42.085659 22473 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.147739 22496 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:42.154122 22496 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.121784 22337 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:42.124249 22066 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:42.124823 22493 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:42.138355 22182 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:42.090543 22450 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:42.166131 22066 raft_consensus.cc:695] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:42.167886 22493 raft_consensus.cc:695] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:42.185341 22493 consensus_queue.cc:237] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:42.179672 20543 connection.cc:232] RPC call timeout handler was delayed by -1.026588022999988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20250831 13:58:42.187446 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:42.170655 22496 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:42.194021 22450 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:42.210520 20646 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:42.212981 22100 consensus_queue.cc:1046] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250831 13:58:42.213577 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1156) took 1686 ms. Trace:
I20250831 13:58:42.213966 22375 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:42.167718 22182 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.215039 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:42.215584 20651 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:42.215696 22489 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:42.216110 22182 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:42.195909 22066 consensus_queue.cc:237] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.216284 22489 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.217382 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1086) took 1181 ms. Trace:
I20250831 13:58:42.217712 20645 rpcz_store.cc:276] 0831 13:58:41.036054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.036602 (+ 548us) service_pool.cc:225] Handling call
0831 13:58:42.217203 (+1180601us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b540030fcd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:42.217299 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98315136}
I20250831 13:58:42.194293 21914 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:42.176764 22365 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:42.204744 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:42.220291 20573 raft_consensus.cc:2391] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:42.207541 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 819) took 1014 ms. Trace:
I20250831 13:58:42.209142 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:42.221218 20642 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:42.220949 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:42.222405 20646 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:42.199458 22473 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:42.220691 20648 rpcz_store.cc:276] 0831 13:58:41.193393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.746572 (+553179us) service_pool.cc:225] Handling call
0831 13:58:41.747002 (+ 430us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:41.751561 (+ 4559us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:41.751590 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:41.752153 (+ 563us) log.cc:844] Serialized 52 byte log entry
0831 13:58:42.159458 (+407305us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:42.159510 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:42.159598 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:42.207311 (+ 47713us) raft_consensus.cc:1683] finished
0831 13:58:42.207334 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:42.207484 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4342,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":227,"thread_start_us":112621,"threads_started":1,"wal-append.queue_time_us":152924}
I20250831 13:58:42.224815 20541 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:42.222971 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:42.223215 20651 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:58:42.213909 22100 consensus_queue.cc:1230] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20250831 13:58:42.218134 22489 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:42.213977 20571 rpcz_store.cc:276] 0831 13:58:40.526739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.527400 (+ 661us) service_pool.cc:225] Handling call
0831 13:58:40.561817 (+ 34417us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:41.253421 (+691604us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:42.213358 (+959937us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54001fbb54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:42.213522 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1588941056}
W20250831 13:58:42.237435 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:42.228305 21708 raft_consensus.cc:3053] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:42.242650 20642 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:42.202438 22229 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.246943 20541 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:42.249277 21174 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:42.250939 22450 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:42.251540 22450 raft_consensus.cc:695] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 6 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:42.252519 22450 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:42.255421 22361 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:42.255956 22361 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:42.257929 22361 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:42.259233 22374 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.259516 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:42.227639 22336 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.262136 22336 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:42.264201 22461 log.cc:927] Time spent T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.683s user 0.003s sys 0.000s
I20250831 13:58:42.243070 21708 raft_consensus.cc:738] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:42.265688 21708 consensus_queue.cc:260] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.267144 21708 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:42.272353 22229 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:42.221463 21914 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.279431 20568 raft_consensus.cc:1273] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:42.280735 22374 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:42.233820 22468 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20250831 13:58:42.229804 22327 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:42.284452 22327 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:42.284904 22327 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:42.285298 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:58:42.291422 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 44583.5us
I20250831 13:58:42.293066 22473 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:42.296977 22493 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:42.297657 22493 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:42.301234 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:42.302733 22493 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:42.301260 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1234) took 1188 ms. Trace:
I20250831 13:58:42.306118 20493 rpcz_store.cc:276] 0831 13:58:41.112891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.260487 (+147596us) service_pool.cc:225] Handling call
0831 13:58:41.260801 (+ 314us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:41.278716 (+ 17915us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:41.278764 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:41.279337 (+ 573us) log.cc:844] Serialized 26 byte log entry
0831 13:58:41.408793 (+129456us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:41.408858 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:41.408952 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:42.301033 (+892081us) raft_consensus.cc:1683] finished
0831 13:58:42.301063 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:42.301204 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17529,"raft.queue_time_us":19446,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":8739,"thread_start_us":100333,"threads_started":1,"wal-append.queue_time_us":337552}
W20250831 13:58:42.311800 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:42.314311 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
W20250831 13:58:42.316470 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:42.324412 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:42.325090 20642 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:42.322759 22473 raft_consensus.cc:695] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:42.355013 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:42.355589 22473 consensus_queue.cc:237] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:42.355662 20500 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:42.364370 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:42.365163 20569 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:42.366942 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:42.367650 20570 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:42.390507 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:42.435120 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:42.449146 20500 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:42.450481 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:42.527242 20493 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:42.681384 20642 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
W20250831 13:58:42.692622 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:42.693637 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 310299us
I20250831 13:58:42.720597 20570 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:42.722198 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:42.723007 20570 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:42.851428 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:42.852336 20500 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
W20250831 13:58:42.897549 21205 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.017s sys 0.078s
I20250831 13:58:42.920172 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:43.244225 20500 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
W20250831 13:58:42.891477 22519 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.408s user 0.000s sys 0.000s
I20250831 13:58:42.991531 20493 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
W20250831 13:58:42.910841 20466 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.003s sys 0.018s
W20250831 13:58:43.032745 20569 consensus_meta.cc:287] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.667s user 0.013s sys 0.012s
I20250831 13:58:43.033290 20499 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:43.046944 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 822) took 1656 ms. Trace:
W20250831 13:58:43.180341 20573 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.932s user 0.004s sys 0.022s
W20250831 13:58:43.251868 20617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.866s user 0.027s sys 0.040s
W20250831 13:58:43.256059 20617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.871s user 0.027s sys 0.040s
I20250831 13:58:43.254112 20569 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:43.254796 20644 rpcz_store.cc:276] 0831 13:58:41.390386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.953134 (+562748us) service_pool.cc:225] Handling call
0831 13:58:41.953409 (+ 275us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:41.979790 (+ 26381us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:41.979823 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:41.980275 (+ 452us) log.cc:844] Serialized 26 byte log entry
0831 13:58:42.230581 (+250306us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:42.230616 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:42.230672 (+ 56us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:43.046638 (+815966us) raft_consensus.cc:1683] finished
0831 13:58:43.046668 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.046856 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26023,"raft.queue_time_us":27566,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":125,"thread_start_us":249856,"threads_started":1,"wal-append.queue_time_us":448886}
W20250831 13:58:43.253839 20466 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.937s user 0.003s sys 0.019s
W20250831 13:58:43.255424 20573 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.007s user 0.005s sys 0.022s
I20250831 13:58:43.264150 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:43.252939 21205 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.936s user 0.017s sys 0.078s
I20250831 13:58:43.274482 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:43.282490 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:43.286422 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:43.287132 20500 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:43.288017 20614 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.849s user 0.003s sys 0.010s
W20250831 13:58:43.288328 20614 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.850s user 0.003s sys 0.010s
W20250831 13:58:43.251324 22496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.940s user 0.033s sys 0.063s
W20250831 13:58:43.320883 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 886154us
W20250831 13:58:43.323707 22468 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.873s user 0.008s sys 0.005s
W20250831 13:58:43.367373 22468 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.930s user 0.008s sys 0.005s
W20250831 13:58:43.304553 22496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.993s user 0.033s sys 0.064s
W20250831 13:58:43.294548 21231 debug-util.cc:398] Leaking SignalData structure 0x7b0800258ca0 after lost signal to thread 19366
I20250831 13:58:43.393126 20500 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
W20250831 13:58:43.403434 20541 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.007s sys 0.008s
W20250831 13:58:43.403925 20541 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.007s sys 0.008s
W20250831 13:58:43.404258 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 709637us
W20250831 13:58:43.409294 20542 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.009s sys 0.007s
W20250831 13:58:43.409667 20542 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.716s user 0.009s sys 0.007s
W20250831 13:58:43.410084 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:43.410496 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:43.410856 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:43.394743 20543 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.153s user 0.003s sys 0.011s
W20250831 13:58:43.412328 20543 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.171s user 0.003s sys 0.011s
W20250831 13:58:43.412716 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 1.17149e+06us
W20250831 13:58:43.417764 20543 connection.cc:232] RPC call timeout handler was delayed by -1.1683632589999888s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20250831 13:58:43.408172 21231 debug-util.cc:398] Leaking SignalData structure 0x7b0800258d40 after lost signal to thread 19369
W20250831 13:58:43.428886 20543 connection.cc:232] RPC call timeout handler was delayed by -1.0990632589999905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20250831 13:58:43.429993 20543 connection.cc:232] RPC call timeout handler was delayed by -1.0171632589999717s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20250831 13:58:43.439484 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.997s user 0.023s sys 0.028s
W20250831 13:58:43.441139 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.999s user 0.023s sys 0.028s
I20250831 13:58:43.427939 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47732 (request call id 138) took 1098 ms. Trace:
W20250831 13:58:43.471006 20615 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.008s sys 0.024s
W20250831 13:58:43.472637 20615 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.882s user 0.008s sys 0.024s
I20250831 13:58:43.464591 22528 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.464392 22535 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.489075 22533 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.489632 22533 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.481750 22537 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.523408 22534 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.553540 22534 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.565327 22493 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.559213 22530 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.550696 22531 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.485633 22528 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.569113 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1250) took 1200 ms. Trace:
I20250831 13:58:43.491585 22535 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:43.575718 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:43.576381 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:43.576835 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:43.567531 22206 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.876s user 0.013s sys 0.017s
W20250831 13:58:43.604060 22206 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.017s user 0.014s sys 0.017s
I20250831 13:58:43.609164 22543 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.516599 22537 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.609702 22543 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.609800 22206 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:43.610532 20451 thread.cc:641] client-negotiator [worker] (thread pool) Time spent creating pthread: real 2.285s user 0.025s sys 0.067s
W20250831 13:58:43.611007 20451 thread.cc:608] client-negotiator [worker] (thread pool) Time spent starting thread: real 2.289s user 0.025s sys 0.067s
I20250831 13:58:43.610906 22206 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.611678 22543 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.577869 22541 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.621948 22541 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.623287 22545 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:43.623806 22545 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.625660 22545 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.581281 22531 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:43.469434 21205 rpcz_store.cc:276] 0831 13:58:42.315236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:42.315403 (+ 167us) service_pool.cc:225] Handling call
0831 13:58:42.315740 (+ 337us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:42.315953 (+ 213us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 13:58:42.316261 (+ 308us) log.cc:844] Serialized 14 byte log entry
0831 13:58:43.323732 (+1007471us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:43.342596 (+ 18864us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:43.388473 (+ 45877us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:43.396961 (+ 8488us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.413809 (+ 16848us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":134,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"spinlock_wait_cycles":5348224,"thread_start_us":958829,"threads_started":1,"wal-append.queue_time_us":969349}
I20250831 13:58:43.614816 22206 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.630065 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1240) took 1540 ms. Trace:
I20250831 13:58:43.630421 20573 rpcz_store.cc:276] 0831 13:58:42.090136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:42.227512 (+137376us) service_pool.cc:225] Handling call
0831 13:58:42.227812 (+ 300us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:42.228593 (+ 781us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:42.228616 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:42.230206 (+ 1590us) log.cc:844] Serialized 26 byte log entry
0831 13:58:43.261129 (+1030923us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:43.261174 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:43.261245 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:43.629878 (+368633us) raft_consensus.cc:1683] finished
0831 13:58:43.629902 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.630012 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1286,"raft.queue_time_us":567,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":7502,"thread_start_us":1012469,"threads_started":1,"wal-append.queue_time_us":1340191}
I20250831 13:58:43.631711 22528 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:43.617475 22549 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:43.618425 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1241) took 1526 ms. Trace:
I20250831 13:58:43.572180 22493 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.573357 22530 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:43.621390 22535 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:43.641076 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1228) took 2554 ms. Trace:
I20250831 13:58:43.641552 20496 rpcz_store.cc:276] 0831 13:58:41.065669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.065827 (+ 158us) service_pool.cc:225] Handling call
0831 13:58:41.066177 (+ 350us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:41.072109 (+ 5932us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:41.072141 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:41.072556 (+ 415us) log.cc:844] Serialized 26 byte log entry
0831 13:58:41.183315 (+110759us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:41.183370 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:41.183453 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:43.619564 (+2436111us) raft_consensus.cc:1683] finished
0831 13:58:43.619591 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.619753 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5551,"raft.queue_time_us":8793,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":93,"thread_start_us":84218,"threads_started":1,"wal-append.queue_time_us":267121,"wal-append.run_cpu_time_us":2848,"wal-append.run_wall_time_us":1365002}
I20250831 13:58:43.643946 22549 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:43.645021 22543 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.645566 22543 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.646826 22493 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.634043 22541 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.647454 22543 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.638672 20571 rpcz_store.cc:276] 0831 13:58:42.091898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:42.232238 (+140340us) service_pool.cc:225] Handling call
0831 13:58:42.232547 (+ 309us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:42.232927 (+ 380us) raft_consensus.cc:1532] Early marking committed up to index 1
0831 13:58:42.232963 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:42.233406 (+ 443us) log.cc:844] Serialized 26 byte log entry
0831 13:58:42.409092 (+175686us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:42.409162 (+ 70us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:42.409252 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:43.618181 (+1208929us) raft_consensus.cc:1683] finished
0831 13:58:43.618209 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.618359 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":102,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":175348,"threads_started":1,"wal-append.queue_time_us":1314897}
I20250831 13:58:43.634105 22542 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:43.569741 20570 rpcz_store.cc:276] 0831 13:58:42.368766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:42.723670 (+354904us) service_pool.cc:225] Handling call
0831 13:58:42.723999 (+ 329us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:42.724223 (+ 224us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:43.413773 (+689550us) log.cc:844] Serialized 14 byte log entry
0831 13:58:43.568442 (+154669us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:43.568557 (+ 115us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:43.568636 (+ 79us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:43.568780 (+ 144us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.569021 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":829061,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"thread_start_us":511019,"threads_started":2}
I20250831 13:58:43.688230 22530 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:43.696408 22545 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:43.696525 22544 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:43.707546 22544 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.709937 22544 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.701622 22551 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.711216 22551 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.712751 22468 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:43.713307 22468 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:43.727190 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 176018us
I20250831 13:58:43.685817 22542 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.756271 20573 raft_consensus.cc:1273] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:43.751950 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1252) took 1369 ms. Trace:
W20250831 13:58:43.762888 20498 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.609s user 0.010s sys 0.012s
W20250831 13:58:43.764253 20498 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.610s user 0.011s sys 0.012s
I20250831 13:58:43.764993 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1231) took 2692 ms. Trace:
I20250831 13:58:43.765393 20498 rpcz_store.cc:276] 0831 13:58:41.072317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.092771 (+ 20454us) service_pool.cc:225] Handling call
0831 13:58:41.093033 (+ 262us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:41.105312 (+ 12279us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:41.105357 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:41.111320 (+ 5963us) log.cc:844] Serialized 26 byte log entry
0831 13:58:43.764630 (+2653310us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:43.764703 (+ 73us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:43.764779 (+ 76us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:43.764825 (+ 46us) raft_consensus.cc:1683] finished
0831 13:58:43.764835 (+ 10us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.764936 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33955,"raft.queue_time_us":7650,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":146,"thread_start_us":2610702,"threads_started":1,"wal-append.queue_time_us":607926,"wal-append.run_cpu_time_us":1598,"wal-append.run_wall_time_us":1082982}
I20250831 13:58:43.730899 22544 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.704990 22545 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:43.759255 20577 rpcz_store.cc:276] 0831 13:58:42.383063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:43.213717 (+830654us) service_pool.cc:225] Handling call
0831 13:58:43.258871 (+ 45154us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:43.259063 (+ 192us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:43.566608 (+307545us) log.cc:844] Serialized 14 byte log entry
0831 13:58:43.751564 (+184956us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:43.751626 (+ 62us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:43.751688 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:43.751767 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.751890 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":294435,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"thread_start_us":466048,"threads_started":2}
I20250831 13:58:43.783850 20500 raft_consensus.cc:1273] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20250831 13:58:43.790190 20500 pending_rounds.cc:85] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 2
I20250831 13:58:43.790419 20500 pending_rounds.cc:107] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20250831 13:58:43.790619 20500 raft_consensus.cc:2887] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:43.738875 22554 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:43.798293 22554 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.788646 22530 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.802495 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:43.802044 22206 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.802872 22530 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.803298 22206 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:43.804761 22517 log.cc:927] Time spent T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602: Append to log took a long time: real 1.354s user 0.005s sys 0.006s
I20250831 13:58:43.804841 22530 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.805073 22206 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.802302 22372 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.806286 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 832) took 1808 ms. Trace:
I20250831 13:58:43.806118 22372 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.806492 20573 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:58:43.805498 20642 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:43.807188 20651 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:58:43.802006 22337 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.807931 22372 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.802146 22541 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.806623 20649 rpcz_store.cc:276] 0831 13:58:41.997743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:42.073168 (+ 75425us) service_pool.cc:225] Handling call
0831 13:58:42.084101 (+ 10933us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:42.084503 (+ 402us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:42.084545 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:42.105737 (+ 21192us) log.cc:844] Serialized 26 byte log entry
0831 13:58:42.245417 (+139680us) raft_consensus.cc:1643] Marking committed up to 7
0831 13:58:42.245462 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:42.245551 (+ 89us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:43.804207 (+1558656us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:43.805971 (+ 1764us) raft_consensus.cc:1683] finished
0831 13:58:43.806005 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:43.806217 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2092,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":966,"spinlock_wait_cycles":154577664,"thread_start_us":120667,"threads_started":1,"wal-append.queue_time_us":232404}
I20250831 13:58:43.808872 22541 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.811082 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:43.811893 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:43.801815 20616 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:43.812469 20569 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:43.813030 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:43.813416 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:43.814122 20569 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:43.811926 20577 raft_consensus.cc:2391] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:43.814364 20616 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:43.815920 20571 raft_consensus.cc:1273] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:43.817838 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:43.818768 20569 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:43.820030 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:43.820698 20569 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:43.811466 20541 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:43.829195 20500 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:58:43.802456 22493 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:43.837011 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:43.839205 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:43.848050 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:43.817283 22544 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.828145 22541 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:43.828018 22560 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:43.824676 22543 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:43.852607 22493 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:43.837684 22530 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:43.867761 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:43.849259 22406 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.096s user 0.001s sys 0.023s
W20250831 13:58:43.834607 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 121206us
I20250831 13:58:43.889722 22543 raft_consensus.cc:695] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:43.870229 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:43.910048 20617 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:43.807958 22337 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.920765 20617 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:43.921813 22372 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:43.941569 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:43.888338 22406 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.135s user 0.001s sys 0.023s
I20250831 13:58:43.867808 22560 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:43.945186 20617 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:43.951500 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:43.949915 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:43.919442 22530 raft_consensus.cc:695] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:43.934859 22206 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 13:58:43.954478 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:43.971379 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:43.972822 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:43.949005 22560 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:43.983132 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:43.950320 22532 raft_consensus.cc:2762] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250831 13:58:43.994570 22532 consensus_queue.cc:1035] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:43.996133 22532 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:43.960208 22496 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:43.953611 22583 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.003865 22583 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.970534 22206 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:44.005074 22206 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:43.980338 22565 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:43.981113 22473 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:43.945657 22372 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:44.007376 22473 raft_consensus.cc:695] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:44.008423 22473 consensus_queue.cc:237] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.981976 22581 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.988723 22566 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:43.990473 22543 consensus_queue.cc:237] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:43.992221 22563 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.016556 22563 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.017220 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.017993 20500 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:44.019060 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.020387 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:43.993701 22554 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.021853 22554 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:44.022431 22554 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.024802 22554 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.025365 22554 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.025946 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52966 (request call id 441) took 3645 ms. Trace:
I20250831 13:58:44.025846 22554 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:44.007467 22372 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:44.043213 22533 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.026458 20495 rpcz_store.cc:276] 0831 13:58:40.380968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.732747 (+351779us) service_pool.cc:225] Handling call
0831 13:58:44.025611 (+3292864us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b54001626d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97c6e5 00007fb9bc9749ba 00007fb9c18d4079 00007fb9baf64644 00007fb9baf645d2 00007fb9baf64501 00007fb9baf6447c 00007fb9baf63732 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:44.025881 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1246436736}
I20250831 13:58:44.044677 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.045323 22531 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:43.988107 22542 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.047619 22542 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:44.009878 22581 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.006990 22565 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.013597 22206 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.015321 22566 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:44.019217 22526 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.733s user 0.000s sys 0.002s
I20250831 13:58:43.984110 22406 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.029997 20572 raft_consensus.cc:3053] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:44.063578 20572 raft_consensus.cc:738] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:44.031162 20500 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:43.992008 22530 consensus_queue.cc:237] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.051759 22545 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:44.060528 22566 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.053417 22544 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:44.055456 22337 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:44.057552 22206 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:44.071075 20572 consensus_queue.cc:260] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.075791 20572 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:44.076545 22563 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.056034 22565 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.057085 22581 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.058454 22537 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.059507 20495 raft_consensus.cc:2408] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:44.051914 22468 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.068960 22583 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.069772 22493 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:44.053418 22551 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.048048 22549 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:44.048135 22542 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.092715 22542 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.094295 22542 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.070559 22406 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:44.096269 20467 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.702s user 0.054s sys 0.094s
I20250831 13:58:44.097199 22406 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.088090 22565 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.099401 22565 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.101812 22565 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.106920 22566 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.107558 22566 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.108223 22429 raft_consensus.cc:3053] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:44.108740 22429 raft_consensus.cc:738] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:44.110016 22566 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.062119 22531 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.111135 22531 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.111984 22335 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:44.112568 22335 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:44.086992 20497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.677s user 0.018s sys 0.007s
I20250831 13:58:44.114085 22531 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:44.043221 22534 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.109648 22429 consensus_queue.cc:260] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.127043 22372 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.084976 20500 raft_consensus.cc:3053] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:44.134752 20500 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:44.135608 20572 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20250831 13:58:44.133967 22335 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.135843 20500 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.144119 20499 raft_consensus.cc:1273] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:44.145119 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1249) took 1713 ms. Trace:
I20250831 13:58:44.145207 20500 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:44.145506 20499 rpcz_store.cc:276] 0831 13:58:42.431985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:43.275988 (+844003us) service_pool.cc:225] Handling call
0831 13:58:44.140156 (+864168us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:44.140398 (+ 242us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:44.144887 (+ 4489us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:44.145070 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16446208}
I20250831 13:58:44.151206 22496 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:44.155236 20572 pending_rounds.cc:85] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 1
I20250831 13:58:44.155575 20572 pending_rounds.cc:107] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20250831 13:58:44.155871 20572 raft_consensus.cc:2887] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:44.157657 20649 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:44.163784 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.164331 20646 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:44.165009 20651 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:44.165100 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.166029 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.166137 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1217) took 2836 ms. Trace:
W20250831 13:58:44.166960 22329 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20250831 13:58:44.166589 20572 rpcz_store.cc:276] 0831 13:58:41.329330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.329470 (+ 140us) service_pool.cc:225] Handling call
0831 13:58:41.329780 (+ 310us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:44.029885 (+2700105us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b54002f7ea0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc905550 00007fb9bc978170 00007fb9bc9791e1 00007fb9bc975d4f 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c
0831 13:58:44.165828 (+135943us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:44.166059 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9430912}
I20250831 13:58:44.167520 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:44.168856 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.171581 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:44.172261 20649 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:44.167191 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:44.176927 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.177585 20648 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:44.179349 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:44.165019 20466 connection.cc:232] RPC call timeout handler was delayed by -1.4101282889999993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
I20250831 13:58:44.182922 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.183444 20466 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:44.183933 20645 raft_consensus.cc:2408] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:44.184492 22335 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:44.184909 22335 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:44.185232 22335 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:44.189157 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.195196 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.200109 20649 raft_consensus.cc:2408] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 13:58:44.203493 22554 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:44.203511 22372 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:44.142146 21914 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.205791 20466 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:44.208978 22575 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
W20250831 13:58:44.210399 20467 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.816s user 0.054s sys 0.094s
W20250831 13:58:44.211515 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 2.81694e+06us
I20250831 13:58:44.134243 22542 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.213649 20467 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:44.213820 20466 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:44.237531 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:44.244168 20496 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:44.249130 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 437730us
I20250831 13:58:44.237423 22548 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:44.222847 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:44.260119 20496 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
W20250831 13:58:44.260459 20497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.850s user 0.019s sys 0.007s
I20250831 13:58:44.263001 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:44.263885 20496 raft_consensus.cc:2374] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6. Re-sending same reply.
I20250831 13:58:44.264920 20500 raft_consensus.cc:1273] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20250831 13:58:44.267942 20500 pending_rounds.cc:85] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 3
I20250831 13:58:44.268114 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.270157 20500 pending_rounds.cc:107] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20250831 13:58:44.272210 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1238) took 3118 ms. Trace:
W20250831 13:58:44.277112 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 324087us
I20250831 13:58:44.283172 20497 rpcz_store.cc:276] 0831 13:58:41.153347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.315301 (+161954us) service_pool.cc:225] Handling call
0831 13:58:41.315688 (+ 387us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:41.339063 (+ 23375us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:41.339107 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:41.351549 (+ 12442us) log.cc:844] Serialized 26 byte log entry
0831 13:58:41.975206 (+623657us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:44.261648 (+2286442us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:44.261708 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:44.261798 (+ 90us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:44.261850 (+ 52us) raft_consensus.cc:1683] finished
0831 13:58:44.261868 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:44.272149 (+ 10281us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22849,"raft.queue_time_us":22976,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":7244,"spinlock_wait_cycles":5228416,"thread_start_us":2851261,"threads_started":1,"wal-append.queue_time_us":250019,"wal-append.run_cpu_time_us":5331,"wal-append.run_wall_time_us":1345822}
I20250831 13:58:44.284554 20617 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:44.285498 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.286199 20497 raft_consensus.cc:2374] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6. Re-sending same reply.
I20250831 13:58:44.287935 20500 raft_consensus.cc:2887] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:44.290051 20500 raft_consensus.cc:1238] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20250831 13:58:44.291604 20496 raft_consensus.cc:2374] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6. Re-sending same reply.
W20250831 13:58:44.300560 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 81354us
I20250831 13:58:44.194445 22335 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.292855 21914 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:44.301349 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 48424.1us
I20250831 13:58:44.301831 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:44.267794 22548 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.318843 20617 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:44.320715 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 105419us
I20250831 13:58:44.307894 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1256) took 1873 ms. Trace:
I20250831 13:58:44.322130 20493 rpcz_store.cc:276] 0831 13:58:42.434146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:43.395004 (+960858us) service_pool.cc:225] Handling call
0831 13:58:43.415869 (+ 20865us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:43.416079 (+ 210us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:43.694713 (+278634us) log.cc:844] Serialized 14 byte log entry
0831 13:58:44.307578 (+612865us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:44.307630 (+ 52us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:44.307668 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:44.307749 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:44.307844 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":385133,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":257120,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":2274,"thread_start_us":856207,"threads_started":3,"wal-append.queue_time_us":193171}
I20250831 13:58:44.299542 22565 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:44.303409 22067 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:44.300779 22591 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:44.332093 22534 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:44.348168 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:44.348883 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:44.295364 22560 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.333499 22542 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.358668 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.358848 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.359287 20647 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:44.309794 22592 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:44.330750 22554 raft_consensus.cc:2762] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20250831 13:58:44.356175 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.356971 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.357092 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.358508 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.370594 22484 consensus_queue.cc:1035] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:44.345366 22591 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:44.309361 21914 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.343533 22067 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:44.373095 20644 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.008s sys 0.016s
I20250831 13:58:44.372473 22549 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:44.342854 22496 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20250831 13:58:44.373466 20493 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:44.376813 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.340008 22565 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.378919 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:44.375919 22587 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.072s user 0.000s sys 0.001s
I20250831 13:58:44.404927 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.403170 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.337222 22559 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
W20250831 13:58:44.384212 20644 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.008s sys 0.016s
I20250831 13:58:44.378082 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:44.373431 22592 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:44.424187 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:44.411617 20543 leader_election.cc:336] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.526s (SENT)
I20250831 13:58:44.384037 22554 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:44.375891 22537 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:44.384511 21708 consensus_meta.cc:287] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 2.117s user 0.020s sys 0.011s
I20250831 13:58:44.433081 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.383219 22542 consensus_queue.cc:1046] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250831 13:58:44.433984 20642 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
W20250831 13:58:44.515975 22066 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.252s user 0.025s sys 0.033s
I20250831 13:58:44.545899 22493 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:44.498909 22327 consensus_meta.cc:287] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 2.214s user 0.021s sys 0.005s
I20250831 13:58:44.580263 20649 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:58:44.580498 22493 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.510313 22335 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.496639 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:44.493340 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:44.470840 22596 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
W20250831 13:58:44.585357 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 281492us
I20250831 13:58:44.573210 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:44.583290 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:44.590216 20648 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:44.434720 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:44.578774 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:44.593427 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.537573 20616 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:44.596622 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:44.580736 22327 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.546730 20617 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:44.547313 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 225786us
I20250831 13:58:44.519497 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.600607 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.602283 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:44.607088 20497 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:44.545137 22595 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:44.617558 22608 log.cc:927] Time spent T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20250831 13:58:44.618099 22582 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.618518 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:44.475891 22604 log.cc:927] Time spent T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
W20250831 13:58:44.621759 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 75161.9us
W20250831 13:58:44.621923 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 84470.6us
I20250831 13:58:44.623026 20466 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:44.623227 20467 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:44.623540 20616 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:44.624856 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:44.475917 22588 log.cc:927] Time spent T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
W20250831 13:58:44.476585 22601 log.cc:927] Time spent T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20250831 13:58:44.465682 22605 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:44.617508 22598 log.cc:927] Time spent T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.091s user 0.001s sys 0.000s
I20250831 13:58:44.628746 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.628966 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.629364 20573 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:44.551873 22541 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:44.574282 22365 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.527509 22067 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:44.646135 20497 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:44.617408 22595 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:44.551977 22358 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.123s
I20250831 13:58:44.734885 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:44.735771 20497 raft_consensus.cc:2374] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 44d163094add49049cc749547d891602 in term 8. Re-sending same reply.
I20250831 13:58:44.435142 22592 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:44.505553 22530 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.630391 22543 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:44.734097 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.737234 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:44.739423 20573 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:44.619203 21708 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.620200 22560 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:44.629520 20571 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:44.740159 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:44.747145 20497 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:44.737648 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.629026 22605 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:44.743501 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 120548us
W20250831 13:58:44.743541 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 114457us
I20250831 13:58:44.471693 22607 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.360s
W20250831 13:58:44.742719 22066 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.479s user 0.032s sys 0.036s
I20250831 13:58:44.742905 22301 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20250831 13:58:44.476028 22586 consensus_queue.cc:1046] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.311s
W20250831 13:58:44.750506 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 138034us
I20250831 13:58:44.751530 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.688823 22100 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:44.743777 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.710453 22531 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:44.758721 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:44.748306 20387 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 90a0a56228d04660b3cb351485be6593 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 58aded50ab0d4609a053647eb11b1f48 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
W20250831 13:58:44.746742 22603 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20250831 13:58:44.608826 22537 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.763418 20466 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:44.764259 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.775353 20571 raft_consensus.cc:2391] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:44.776341 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.777549 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.739574 22554 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:44.766889 20573 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:44.764277 20575 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:44.768074 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.740821 22543 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:44.690623 22365 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.618603 22582 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.780494 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.785051 20497 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:44.818562 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.827997 20500 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:44.828337 20571 raft_consensus.cc:2408] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 13:58:44.828820 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:44.829341 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.829418 20500 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:44.798771 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.830431 22327 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:44.831023 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 67703.8us
I20250831 13:58:44.831166 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.834581 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.835211 20571 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 2.
I20250831 13:58:44.836030 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:44.829604 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:44.801836 20646 raft_consensus.cc:1273] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:44.801155 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.849758 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:44.853590 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.760495 22484 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 13:58:44.779054 22594 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.326s user 0.000s sys 0.001s
I20250831 13:58:44.758751 22614 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:44.743052 22530 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.857257 20570 raft_consensus.cc:2391] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:44.870306 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.748059 21708 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:44.857962 20646 raft_consensus.cc:2391] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:44.752158 22605 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:44.764959 22560 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:44.778947 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47746 (request call id 74) took 1338 ms. Trace:
I20250831 13:58:44.859333 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:44.861680 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.863529 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52966 (request call id 443) took 3933 ms. Trace:
I20250831 13:58:44.856343 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:44.771848 22531 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:44.773819 22100 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:44.851634 22543 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:44.764787 22335 consensus_queue.cc:1046] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
I20250831 13:58:44.820539 22554 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:44.859552 22484 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:44.859299 22493 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:44.861459 22581 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.868570 22614 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:44.883531 20494 rpcz_store.cc:276] 0831 13:58:40.920349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.920504 (+ 155us) service_pool.cc:225] Handling call
0831 13:58:44.853569 (+3933065us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b54001626d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8facd0 00007fb9bc97eeea 00007fb9c18d760e 00007fb9baf74cc7 00007fb9baf74c52 00007fb9baf74b81 00007fb9baf74afc 00007fb9baf73db2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:44.854041 (+ 472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":924250496}
I20250831 13:58:44.882869 20573 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:44.886426 20645 raft_consensus.cc:2391] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
I20250831 13:58:44.926630 20577 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:44.875438 22560 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:44.938925 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.897037 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.924348 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.923669 20647 raft_consensus.cc:1273] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250831 13:58:44.882892 20541 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:44.911162 20646 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:44.881088 22593 consensus_queue.cc:1046] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20250831 13:58:44.886543 22531 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:44.885193 22566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.008s sys 0.057s
W20250831 13:58:44.948434 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 65643us
I20250831 13:58:45.038251 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.949085 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.039290 20571 raft_consensus.cc:2366] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6.
I20250831 13:58:44.952340 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.039575 20575 raft_consensus.cc:2408] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20250831 13:58:45.040222 20493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s user 0.003s sys 0.011s
I20250831 13:58:44.964602 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1239) took 2884 ms. Trace:
W20250831 13:58:45.040522 20493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s user 0.003s sys 0.011s
I20250831 13:58:44.910496 22067 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:45.039556 20541 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:45.040716 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:45.042559 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 417776us
I20250831 13:58:45.044183 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.040998 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.039669 20569 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:44.927774 22484 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:44.934880 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 61433.8us
I20250831 13:58:44.935676 22614 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:45.045859 20575 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:44.953814 22599 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.363s user 0.000s sys 0.008s
I20250831 13:58:45.044852 20616 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:45.065517 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:44.964715 22582 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.048981 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:44.961947 22365 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.051707 22066 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:44.958038 22188 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:45.057140 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:45.057238 22361 consensus_queue.cc:1046] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:45.045346 20577 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:44.959195 22610 log.cc:927] Time spent T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.200s user 0.000s sys 0.000s
W20250831 13:58:45.040910 22182 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.005s sys 0.009s
W20250831 13:58:45.041169 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 374974us
I20250831 13:58:45.088544 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:44.955178 22551 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:58:44.923794 22554 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:44.938736 22560 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:45.047461 22566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.710s user 0.016s sys 0.058s
I20250831 13:58:45.057024 22537 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.064160 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:45.103122 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.068120 20573 raft_consensus.cc:2408] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 13:58:44.910388 22607 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250831 13:58:44.963963 22530 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.070742 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.107254 20569 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:44.966253 22595 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:45.072721 20616 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:45.095412 22468 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:44.953667 22606 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.333s
I20250831 13:58:45.050100 22613 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:45.075667 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.053961 22067 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:45.040728 20574 rpcz_store.cc:276] 0831 13:58:42.080365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:42.126611 (+ 46246us) service_pool.cc:225] Handling call
0831 13:58:42.126995 (+ 384us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:42.128441 (+ 1446us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:42.128478 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:42.128987 (+ 509us) log.cc:844] Serialized 26 byte log entry
0831 13:58:44.527233 (+2398246us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b1800262ba0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171 00007fb9bc94909f
0831 13:58:44.675967 (+148734us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:44.676111 (+ 144us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:44.680114 (+ 4003us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:44.964229 (+284115us) raft_consensus.cc:1683] finished
0831 13:58:44.964260 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:44.964530 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9013,"raft.queue_time_us":1086,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":1985,"spinlock_wait_cycles":20584832,"thread_start_us":148260,"threads_started":1,"wal-append.queue_time_us":2745823}
W20250831 13:58:45.083024 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:45.116557 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:45.116979 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.116729 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.074180 22066 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:45.084528 22361 raft_consensus.cc:3053] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:58:45.084954 21210 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:45.098073 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.063344 22473 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
W20250831 13:58:45.087634 22182 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.006s sys 0.009s
I20250831 13:58:45.079425 22493 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:44.935003 22581 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.104085 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.044034 21708 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:45.109433 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.111380 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.112165 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.130157 20577 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:45.113628 20493 raft_consensus.cc:1273] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:45.074254 22188 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.098258 22551 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:45.121253 22067 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:45.138155 20569 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:45.119489 22361 raft_consensus.cc:738] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:45.129150 22493 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:45.143883 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.119681 22066 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:45.194134 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.194506 20573 raft_consensus.cc:2408] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:45.194286 20617 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:45.194805 20494 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:58:45.163887 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.197232 20571 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250831 13:58:45.198397 20466 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:45.199826 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.200421 20493 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:45.115772 22468 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.143774 22530 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:45.153055 22551 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:45.209043 20569 raft_consensus.cc:1273] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:45.196424 22493 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:45.231254 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 117125us
I20250831 13:58:45.229082 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.233379 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.234012 20494 raft_consensus.cc:2366] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:58:44.881923 21208 rpcz_store.cc:276] 0831 13:58:43.456932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:43.457088 (+ 156us) service_pool.cc:225] Handling call
0831 13:58:43.457567 (+ 479us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:43.690616 (+233049us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:43.693424 (+ 2808us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:43.883281 (+189857us) log.cc:844] Serialized 26 byte log entry
0831 13:58:44.201349 (+318068us) raft_consensus.cc:1643] Marking committed up to 7
0831 13:58:44.204700 (+ 3351us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:44.204803 (+ 103us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:44.761196 (+556393us) raft_consensus.cc:1683] finished
0831 13:58:44.764890 (+ 3694us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:44.778872 (+ 13982us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":340454,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":485314,"threads_started":2,"wal-append.queue_time_us":350232}
I20250831 13:58:45.229692 22188 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.240942 20616 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:45.246876 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.129283 22613 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.249105 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.230031 22636 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:45.231835 22640 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:45.248885 22188 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20250831 13:58:45.257751 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.258528 22631 raft_consensus.cc:3053] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:45.263314 22631 raft_consensus.cc:738] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:45.264374 22631 consensus_queue.cc:260] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.265693 22631 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:45.267412 22628 log.cc:927] Time spent T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20250831 13:58:45.267938 22644 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
W20250831 13:58:45.268463 22635 log.cc:927] Time spent T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250831 13:58:45.268641 22644 raft_consensus.cc:695] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:45.269491 22629 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20250831 13:58:45.269842 22644 consensus_queue.cc:237] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.271049 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:45.271050 22627 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20250831 13:58:45.301388 20500 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:45.271368 22643 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:45.250809 22586 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 13:58:45.279544 22632 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20250831 13:58:45.283340 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:45.283833 22468 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:45.249964 22624 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20250831 13:58:45.232748 22530 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.323007 22530 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.252204 22595 consensus_queue.cc:1046] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.045s
I20250831 13:58:45.324528 22595 raft_consensus.cc:3053] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:45.299176 22365 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:45.325034 22595 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:45.308825 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.327756 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.328751 20500 raft_consensus.cc:2408] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 13:58:45.329634 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.309764 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:45.330971 20645 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.331295 22581 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:45.334796 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 140592us
I20250831 13:58:45.306047 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.259169 22636 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.311707 22613 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.335526 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:45.337460 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.340173 20500 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:45.341373 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.341625 20617 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:45.341670 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 100812us
I20250831 13:58:45.342043 20500 raft_consensus.cc:2374] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5. Re-sending same reply.
I20250831 13:58:45.343020 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.243939 22361 consensus_queue.cc:260] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:45.342772 20616 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:45.345011 20616 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:45.345594 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.308207 22643 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:45.326148 22595 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.377768 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 818) took 4197 ms. Trace:
I20250831 13:58:45.379344 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.379359 20650 rpcz_store.cc:276] 0831 13:58:41.180387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.649560 (+469173us) service_pool.cc:225] Handling call
0831 13:58:41.704035 (+ 54475us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:41.935938 (+231903us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:41.936196 (+ 258us) log.cc:844] Serialized 14 byte log entry
0831 13:58:42.106604 (+170408us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:42.106671 (+ 67us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:42.106744 (+ 73us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:45.377434 (+3270690us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b540006a254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:45.377522 (+ 88us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:45.377681 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2482,"prepare.queue_time_us":481,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":149,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":5202,"spinlock_wait_cycles":532821248,"thread_start_us":169575,"threads_started":1,"wal-append.queue_time_us":372187,"wal-append.run_cpu_time_us":1165,"wal-append.run_wall_time_us":1001181}
I20250831 13:58:45.385142 20645 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:45.385427 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.386377 20541 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:45.387365 22468 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:45.387507 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:45.387871 22468 raft_consensus.cc:695] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:45.388113 22648 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:45.388563 22648 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:45.388967 22648 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.388696 22468 consensus_queue.cc:237] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.389679 20616 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:45.390342 21914 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.391006 22646 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:58:45.394127 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:45.396196 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:45.397001 20645 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:45.398183 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:45.399925 20541 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:45.404929 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:45.386016 20500 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:45.373607 22643 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:45.409644 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:45.408571 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.411981 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.418208 20645 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250831 13:58:45.406190 22595 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:45.406038 21914 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:45.425494 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:45.429517 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:45.426067 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 81021.5us
I20250831 13:58:45.434260 22066 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.435341 22365 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:45.437089 22066 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.427096 21708 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
I20250831 13:58:45.439424 21914 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:45.439826 21914 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.440397 22650 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:45.440814 22650 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:45.441148 22650 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:45.445086 22493 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.414997 22361 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:45.450135 22636 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:45.450353 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 60737.2us
I20250831 13:58:45.428864 22647 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:45.475385 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:45.484364 22653 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:45.484078 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.497509 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:45.446555 22554 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.507246 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:45.507894 20650 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:45.508237 20577 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 13:58:45.480965 22647 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:45.513526 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:45.537858 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.541205 20650 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:58:45.542773 20541 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:45.545390 20494 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:45.548048 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:45.514768 21231 debug-util.cc:398] Leaking SignalData structure 0x7b08002dd8c0 after lost signal to thread 20434
I20250831 13:58:45.554106 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.502362 22493 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:45.535524 22592 consensus_meta.cc:287] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.786s user 0.011s sys 0.002s
I20250831 13:58:45.493875 22551 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.560295 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.561034 20575 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:45.562742 20575 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20250831 13:58:45.554829 20467 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:45.555064 20650 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
W20250831 13:58:45.536090 22543 consensus_meta.cc:287] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.598s user 0.010s sys 0.001s
I20250831 13:58:45.565198 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1232) took 4470 ms. Trace:
W20250831 13:58:45.568202 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 92928.6us
I20250831 13:58:45.566892 20575 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:45.571812 22636 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:45.572244 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:45.555621 21176 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.573688 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.515827 22647 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:45.567960 20502 rpcz_store.cc:276] 0831 13:58:41.082283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:41.131562 (+ 49279us) service_pool.cc:225] Handling call
0831 13:58:41.131915 (+ 353us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:41.163930 (+ 32015us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:41.163974 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:41.164409 (+ 435us) log.cc:844] Serialized 26 byte log entry
0831 13:58:41.357426 (+193017us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:41.357506 (+ 80us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:41.357613 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:45.552165 (+4194552us) raft_consensus.cc:1683] finished
0831 13:58:45.552194 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:45.552377 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47493,"raft.queue_time_us":47844,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":10339,"thread_start_us":145842,"threads_started":1,"wal-append.queue_time_us":276397,"wal-append.run_cpu_time_us":2616,"wal-append.run_wall_time_us":1472726}
I20250831 13:58:45.574443 22335 consensus_queue.cc:1046] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20250831 13:58:45.578480 22335 raft_consensus.cc:3053] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:45.578694 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.578857 22335 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:45.579339 20502 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:45.579643 22335 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.580163 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.580736 22335 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.581035 20502 raft_consensus.cc:2408] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20250831 13:58:45.565037 22605 consensus_meta.cc:287] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.680s user 0.009s sys 0.002s
I20250831 13:58:45.582260 20494 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:45.551529 21708 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:58:45.582885 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.587204 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.587917 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.588667 20502 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:58:45.556931 22484 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:45.589779 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.591385 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.518832 21914 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.589637 22606 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:45.595517 20617 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:45.596710 20467 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.613174 20467 leader_election.cc:400] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6.
W20250831 13:58:45.610647 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.573099 22646 raft_consensus.cc:695] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:45.613452 20467 leader_election.cc:403] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20250831 13:58:45.582901 22586 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:45.581200 22357 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:58:45.615953 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 146328us
I20250831 13:58:45.620564 20649 raft_consensus.cc:1238] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:45.624908 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.559182 22614 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.629084 20467 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:45.595345 22566 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:45.611698 21708 raft_consensus.cc:695] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:45.568907 22592 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.575937 22543 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.585554 22067 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.651983 20493 raft_consensus.cc:2408] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250831 13:58:45.638401 20467 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.622567 22649 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:45.659029 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:45.659629 22649 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:45.642314 21914 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.647666 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 776) took 5801 ms. Trace:
I20250831 13:58:45.660158 22649 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:45.660908 20501 rpcz_store.cc:276] 0831 13:58:39.846135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:40.179954 (+333819us) service_pool.cc:225] Handling call
0831 13:58:40.180257 (+ 303us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:41.256204 (+1075947us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:45.647415 (+4391211us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001c0f54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:45.647607 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2473317760}
I20250831 13:58:45.648833 21708 consensus_queue.cc:237] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.664399 21914 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:45.664889 21914 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.651232 20500 raft_consensus.cc:2408] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20250831 13:58:45.660123 22631 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:45.666396 22631 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:45.666913 22631 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.667284 21914 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:45.654098 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 58871.5us
W20250831 13:58:45.639395 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.658483 22595 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:45.591641 22605 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.668562 20617 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:45.670233 22605 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.662540 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.669519 22649 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:45.671772 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.672719 22649 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:45.672993 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.673707 20501 raft_consensus.cc:2374] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6. Re-sending same reply.
I20250831 13:58:45.674252 22649 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:45.674659 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.675945 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.677136 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.678273 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:45.682941 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.682889 20501 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:45.684006 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.680596 20467 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.684221 20642 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:45.633410 22638 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:45.634580 22586 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:45.685848 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.685983 20467 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.686726 20646 raft_consensus.cc:1238] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:45.639328 22658 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:45.688112 20574 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 4: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:45.689563 20467 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.688434 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.691040 20467 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:45.691826 20467 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.695123 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.696449 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.697997 20501 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 3: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:45.687151 22605 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:45.701478 22657 raft_consensus.cc:2762] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250831 13:58:45.690820 22595 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:45.703769 22660 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:45.692797 22566 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:45.704944 22638 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 13:58:45.720357 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.695739 22606 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:45.738431 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.681520 22648 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.696816 22554 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:45.743374 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 125952us
I20250831 13:58:45.705965 22658 raft_consensus.cc:695] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:45.739492 20501 raft_consensus.cc:2391] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:45.751391 22654 log.cc:927] Time spent T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.207s user 0.000s sys 0.002s
I20250831 13:58:45.738607 22551 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.739498 22660 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:45.740178 22606 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:45.758399 22660 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:45.761050 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.742002 22641 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:45.740962 22605 consensus_queue.cc:1046] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20250831 13:58:45.744158 22484 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:45.740916 22595 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20250831 13:58:45.767513 22543 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.768149 22653 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:45.788175 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.788817 20501 raft_consensus.cc:2374] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7. Re-sending same reply.
I20250831 13:58:45.747815 22566 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20250831 13:58:45.768852 22614 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.762259 22606 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:45.768502 22388 raft_consensus.cc:3053] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:45.807003 22388 raft_consensus.cc:738] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:45.797539 22636 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:45.775163 22648 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.707125 22646 consensus_queue.cc:237] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.805444 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:45.806178 20576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1235) took 3734 ms (client timeout 1713 ms). Trace:
W20250831 13:58:45.813683 20576 rpcz_store.cc:269] 0831 13:58:42.071600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.806015 (+3734415us) service_pool.cc:212] Skipping call since client already timed out
0831 13:58:45.806119 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 13:58:45.810370 22388 consensus_queue.cc:260] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.790613 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.816143 20501 raft_consensus.cc:2391] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:45.815856 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:45.817802 22648 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:45.823093 22388 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:45.802078 22631 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.791015 22067 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:45.828776 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 122547us
I20250831 13:58:45.826656 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.830736 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.832178 20576 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.837121 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.837891 20577 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:45.841305 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.841555 20467 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.824885 22592 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.845345 20467 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:45.851583 20502 raft_consensus.cc:2408] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:45.854306 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.858412 20576 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:45.859555 20616 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:45.861443 20467 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:45.863961 22641 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:45.866189 22592 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:45.866629 22335 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:45.867115 22335 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:45.867480 22335 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:45.869339 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.870093 20501 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:45.870790 22614 consensus_queue.cc:1046] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:45.845604 22554 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:45.875370 20501 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:45.876125 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.849767 22631 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.829058 22658 consensus_queue.cc:237] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:45.879045 22531 consensus_meta.cc:287] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.819s user 0.021s sys 0.000s
I20250831 13:58:45.879401 22531 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.881388 20651 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250831 13:58:45.865352 22662 log.cc:927] Time spent T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20250831 13:58:45.883896 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.880609 22643 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.866685 22592 raft_consensus.cc:695] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:45.893636 20501 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:45.895929 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.896867 20577 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:45.898017 20616 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:45.899574 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:45.898627 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 229215us
I20250831 13:58:45.850625 22647 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.898350 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.900897 22338 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:45.901657 22338 raft_consensus.cc:695] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:45.902166 22647 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.873991 22650 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:45.902580 22338 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:45.904299 22650 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:45.905992 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.900359 20576 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:45.908712 20501 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:45.908823 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.909809 20616 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:45.892024 22531 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:45.913899 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.915675 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.913767 22647 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:45.928162 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1293) took 1347 ms. Trace:
I20250831 13:58:45.928547 20497 rpcz_store.cc:276] 0831 13:58:44.581106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.835136 (+254030us) service_pool.cc:225] Handling call
0831 13:58:44.835512 (+ 376us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:44.935542 (+100030us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:44.935586 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:44.947812 (+ 12226us) log.cc:844] Serialized 26 byte log entry
0831 13:58:45.578933 (+631121us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b18004db720. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171 00007fb9bc94909f
0831 13:58:45.634920 (+ 55987us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:45.763360 (+128440us) log.cc:844] Serialized 14 byte log entry
0831 13:58:45.763598 (+ 238us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:45.763672 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:45.917094 (+153422us) raft_consensus.cc:1683] finished
0831 13:58:45.927966 (+ 10872us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:45.928106 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80069,"prepare.queue_time_us":214067,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":2529,"raft.queue_time_us":99675,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":9146,"spinlock_wait_cycles":39282048,"thread_start_us":279106,"threads_started":3,"wal-append.queue_time_us":801081}
I20250831 13:58:45.896304 22335 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:45.911602 22560 consensus_meta.cc:287] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.801s user 0.007s sys 0.008s
W20250831 13:58:45.916464 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:45.932730 20616 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.130:44629 timed out after 1.743s (SENT) [suppressed 19 similar messages]
I20250831 13:58:45.915460 22609 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:45.922461 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:45.933142 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.935453 20497 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:45.927105 20576 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:45.944202 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.945690 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:45.947055 20497 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:45.948174 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.948843 20497 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.914960 22638 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:45.932350 22335 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:45.958385 20576 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:45.928184 22647 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:45.964987 20467 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:45.964761 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.965941 20576 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:45.966099 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.971405 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.971953 20577 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.971693 20576 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:45.973243 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:45.973738 20467 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:45.971683 20497 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:45.974927 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.975729 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.976110 20497 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.983264 20497 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:45.984170 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:45.966305 22647 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.934839 22609 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.030362 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:46.034340 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:46.041752 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:45.967268 22612 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:45.973958 20576 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:45.940268 22560 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.077162 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:46.079701 20497 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:45.967391 22660 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:46.082350 22671 log.cc:927] Time spent T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.141s user 0.000s sys 0.000s
I20250831 13:58:46.082518 22643 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:46.084154 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:46.085199 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:45.983428 22335 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:46.088900 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:46.112334 20575 raft_consensus.cc:2391] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:46.079208 21231 debug-util.cc:398] Leaking SignalData structure 0x7b0800299640 after lost signal to thread 20526
I20250831 13:58:46.076421 22586 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629) with higher term. Message: Invalid argument: T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6.
I20250831 13:58:46.150074 22335 raft_consensus.cc:695] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:46.151304 22335 consensus_queue.cc:237] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.155644 20617 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:45.965744 22638 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.157117 20617 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:46.158430 20617 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:46.127424 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:46.104707 20577 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:58:46.110146 21176 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.143888 22609 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:46.144230 22560 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:46.144902 20650 raft_consensus.cc:1238] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:45.967327 22592 consensus_queue.cc:237] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.147953 22606 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.117585 22612 raft_consensus.cc:695] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:46.162534 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:46.083410 22660 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:46.173413 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:46.285602 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.138090 20497 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:46.286870 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:46.287964 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:46.289106 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:46.225697 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:46.231065 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:46.249696 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 91355.5us
I20250831 13:58:46.293084 20617 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:58:46.282670 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 120265us
I20250831 13:58:46.163758 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:46.301358 20617 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:46.302423 20576 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:46.302717 20617 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:46.249511 22650 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 13:58:46.311708 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.304327 20501 raft_consensus.cc:1215] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2] Dedup: 2.2->[]
I20250831 13:58:46.312307 20501 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 2. Current term is 4. Ops: [2.2-2.2]
I20250831 13:58:46.307814 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:46.313364 20497 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:46.281334 22531 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:46.281152 22431 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 13:58:46.286883 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 354158us
I20250831 13:58:46.317169 20577 raft_consensus.cc:1273] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:46.282171 22609 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:46.344969 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.316342 22229 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:46.316193 22342 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:46.403764 20577 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
W20250831 13:58:46.321154 21231 debug-util.cc:398] Leaking SignalData structure 0x7b0800330d80 after lost signal to thread 20601
I20250831 13:58:46.311961 22650 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:46.319209 22560 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:46.433040 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:46.371407 22431 raft_consensus.cc:695] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:46.426535 22342 raft_consensus.cc:695] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:46.438562 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.442651 22631 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:46.443222 22631 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.443801 22606 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:46.444634 22606 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:46.445355 22631 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:46.445250 22606 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:46.446827 22622 log.cc:927] Time spent T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 1.419s user 0.004s sys 0.013s
W20250831 13:58:46.447162 22661 log.cc:927] Time spent T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.743s user 0.001s sys 0.009s
I20250831 13:58:46.443233 22342 consensus_queue.cc:237] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:46.452657 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.432389 22614 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:46.465315 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:46.373585 22609 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.428613 22650 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:46.471097 21207 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47742 (request call id 79) took 5554 ms (client timeout 1986 ms). Trace:
W20250831 13:58:46.473325 21207 rpcz_store.cc:269] 0831 13:58:40.937859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.466301 (+5528442us) service_pool.cc:212] Skipping call since client already timed out
0831 13:58:46.466459 (+ 158us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250831 13:58:46.473883 22606 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:46.404350 22612 consensus_queue.cc:237] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.481293 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 889) took 1397 ms. Trace:
I20250831 13:58:46.481802 20645 rpcz_store.cc:276] 0831 13:58:45.083587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.420961 (+337374us) service_pool.cc:225] Handling call
0831 13:58:45.421279 (+ 318us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:45.421542 (+ 263us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:45.421596 (+ 54us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:45.422142 (+ 546us) log.cc:844] Serialized 26 byte log entry
0831 13:58:45.614369 (+192227us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:45.614404 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:45.614472 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:46.446597 (+832125us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:46.478424 (+ 31827us) raft_consensus.cc:1683] finished
0831 13:58:46.478455 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:46.481218 (+ 2763us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5781248,"thread_start_us":149276,"threads_started":1,"wal-append.queue_time_us":253044}
I20250831 13:58:46.484226 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:46.475239 22614 raft_consensus.cc:695] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:46.476830 22606 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:46.496898 21210 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 13:58:46.489542 22372 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:46.488111 22638 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:46.496497 22658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.046s sys 0.129s
I20250831 13:58:46.502445 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:46.497061 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:46.633637 20577 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:46.652531 21207 raft_consensus.cc:2408] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20250831 13:58:46.652935 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:46.502628 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:46.638164 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1199) took 1683 ms. Trace:
I20250831 13:58:46.640991 21209 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:46.657498 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 878) took 1857 ms. Trace:
I20250831 13:58:46.657096 20573 rpcz_store.cc:276] 0831 13:58:44.954138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.109096 (+154958us) service_pool.cc:225] Handling call
0831 13:58:46.637562 (+1528466us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b54003d4ad4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:46.637900 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":53743488}
I20250831 13:58:46.657903 20642 rpcz_store.cc:276] 0831 13:58:44.800328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.800446 (+ 118us) service_pool.cc:225] Handling call
0831 13:58:44.800750 (+ 304us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:45.684843 (+884093us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.656992 (+972149us) spinlock_profiling.cc:243] Waited 883 ms on lock 0x7b540028c154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.657289 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2031435136}
I20250831 13:58:46.659716 20573 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:46.659924 22100 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:46.659837 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 824) took 1450 ms. Trace:
I20250831 13:58:46.647544 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
W20250831 13:58:46.649297 21176 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:46.495613 21231 debug-util.cc:398] Leaking SignalData structure 0x7b08001cea20 after lost signal to thread 20675
I20250831 13:58:46.663199 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 825) took 1452 ms. Trace:
W20250831 13:58:46.663595 21231 debug-util.cc:398] Leaking SignalData structure 0x7b0800366380 after lost signal to thread 20676
I20250831 13:58:46.663936 20497 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:46.664984 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 840) took 1316 ms. Trace:
I20250831 13:58:46.665450 20500 rpcz_store.cc:276] 0831 13:58:45.380061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.454536 (+ 74475us) service_pool.cc:225] Handling call
0831 13:58:46.664475 (+1209939us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b540021d9d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:46.664747 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":367398656}
I20250831 13:58:46.502254 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:46.663239 20616 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:46.667065 21202 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:46.667553 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.668211 21202 raft_consensus.cc:2408] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20250831 13:58:46.668562 20643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1115) took 2486 ms (client timeout 3081 ms). Trace:
I20250831 13:58:46.649300 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
W20250831 13:58:46.669019 20643 rpcz_store.cc:269] 0831 13:58:44.182510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.185606 (+ 3096us) service_pool.cc:225] Handling call
0831 13:58:46.667161 (+2481555us) spinlock_profiling.cc:243] Waited 3.87 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:46.668462 (+ 1301us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8905984}
I20250831 13:58:46.669497 20616 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:46.671938 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 850) took 2980 ms. Trace:
I20250831 13:58:46.673537 20616 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:46.672401 20644 rpcz_store.cc:276] 0831 13:58:43.691227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:43.691363 (+ 136us) service_pool.cc:225] Handling call
0831 13:58:43.693279 (+ 1916us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:43.714391 (+ 21112us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:43.828007 (+113616us) log.cc:844] Serialized 14 byte log entry
0831 13:58:44.500819 (+672812us) log.cc:844] Serialized 14 byte log entry
0831 13:58:44.538141 (+ 37322us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:44.538209 (+ 68us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:44.538261 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.671341 (+2133080us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b54001ee454. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.671418 (+ 77us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:46.671743 (+ 325us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":109864,"prepare.queue_time_us":344770,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":73,"raft.queue_time_us":37152,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":34412,"spinlock_wait_cycles":47961984,"thread_start_us":704880,"threads_started":2,"wal-append.queue_time_us":589055,"wal-append.run_cpu_time_us":1810,"wal-append.run_wall_time_us":1084603}
I20250831 13:58:46.666458 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:46.660880 20502 rpcz_store.cc:276] 0831 13:58:45.209087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.589348 (+380261us) service_pool.cc:225] Handling call
0831 13:58:46.659362 (+1070014us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b540035acd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:46.659658 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":599601920}
I20250831 13:58:46.675788 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1123) took 1827 ms. Trace:
I20250831 13:58:46.678351 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1193) took 1725 ms. Trace:
I20250831 13:58:46.679873 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1121) took 1845 ms. Trace:
I20250831 13:58:46.681290 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 881) took 1880 ms. Trace:
I20250831 13:58:46.663710 20501 rpcz_store.cc:276] 0831 13:58:45.210139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.911241 (+701102us) service_pool.cc:225] Handling call
0831 13:58:45.934142 (+ 22901us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:46.318862 (+384720us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.662591 (+343729us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54001865d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.662974 (+ 383us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":851133696}
I20250831 13:58:46.682049 20616 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:46.716024 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:46.690918 20502 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:46.699875 20571 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
W20250831 13:58:46.694898 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:46.721096 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.699636 20500 raft_consensus.cc:3053] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
W20250831 13:58:46.700062 21231 debug-util.cc:398] Leaking SignalData structure 0x7b0800372f40 after lost signal to thread 21234
I20250831 13:58:46.721742 20500 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:46.722004 21231 debug-util.cc:398] Leaking SignalData structure 0x7b0800372f00 after lost signal to thread 21235
I20250831 13:58:46.723428 20500 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.725245 20500 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:46.726291 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1206) took 1219 ms. Trace:
I20250831 13:58:46.685662 20574 rpcz_store.cc:276] 0831 13:58:44.952397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.116507 (+164110us) service_pool.cc:225] Handling call
0831 13:58:46.677909 (+1561402us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b54000b4fd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:46.678158 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1312071936}
I20250831 13:58:46.715476 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:46.714119 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:46.715062 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1243) took 4594 ms. Trace:
I20250831 13:58:46.686039 20646 rpcz_store.cc:276] 0831 13:58:44.834517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.039749 (+205232us) service_pool.cc:225] Handling call
0831 13:58:45.040042 (+ 293us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:45.687893 (+647851us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.679293 (+991400us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b5400371754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.679675 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1486821248}
I20250831 13:58:46.731679 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 879) took 1931 ms. Trace:
I20250831 13:58:46.732774 20643 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:46.727713 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 45687.1us
I20250831 13:58:46.732158 20651 rpcz_store.cc:276] 0831 13:58:44.800590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.802630 (+ 2040us) service_pool.cc:225] Handling call
0831 13:58:44.803004 (+ 374us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:45.881876 (+1078872us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.731038 (+849162us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540029f254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.731477 (+ 439us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2479645568}
I20250831 13:58:46.732674 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:46.686390 20647 rpcz_store.cc:276] 0831 13:58:44.801096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.817220 (+ 16124us) service_pool.cc:225] Handling call
0831 13:58:44.825207 (+ 7987us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:44.946729 (+121522us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.680752 (+1734023us) spinlock_profiling.cc:243] Waited 86.2 ms on lock 0x7b540028ad54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.681075 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198303616}
W20250831 13:58:46.733722 20494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 831) took 1489 ms (client timeout 1502 ms). Trace:
I20250831 13:58:46.733686 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:46.732609 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:46.685283 20649 rpcz_store.cc:276] 0831 13:58:44.848293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.058654 (+210361us) service_pool.cc:225] Handling call
0831 13:58:45.058984 (+ 330us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:45.621062 (+562078us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.675225 (+1054163us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b5400371254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.675577 (+ 352us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1291046272}
I20250831 13:58:46.656659 20577 raft_consensus.cc:1238] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:46.741472 20644 raft_consensus.cc:2391] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:58:46.739040 20494 rpcz_store.cc:269] 0831 13:58:45.244117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.247708 (+ 3591us) service_pool.cc:225] Handling call
0831 13:58:46.733365 (+1485657us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b540013d654. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:46.733539 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":660118272}
I20250831 13:58:46.742759 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.741292 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.743858 20651 raft_consensus.cc:2408] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:46.731940 20568 rpcz_store.cc:276] 0831 13:58:42.120731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:42.263749 (+143018us) service_pool.cc:225] Handling call
0831 13:58:42.264124 (+ 375us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:42.279746 (+ 15622us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.714337 (+4434591us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b54003b4a54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.714802 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34451328}
I20250831 13:58:46.744848 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:46.734635 21708 consensus_queue.cc:1046] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20250831 13:58:46.746227 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:46.747423 20568 raft_consensus.cc:1238] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 2. Current term is 3. Ops: []
I20250831 13:58:46.750537 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.734611 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1127) took 1536 ms. Trace:
I20250831 13:58:46.751272 20650 rpcz_store.cc:276] 0831 13:58:45.197862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.563847 (+365985us) service_pool.cc:225] Handling call
0831 13:58:45.564110 (+ 263us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:46.250631 (+686521us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.734152 (+483521us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b540028b254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.734422 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1335313792}
I20250831 13:58:46.752195 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.735893 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1268) took 2143 ms. Trace:
I20250831 13:58:46.752978 20572 rpcz_store.cc:276] 0831 13:58:44.592188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.592346 (+ 158us) service_pool.cc:225] Handling call
0831 13:58:44.592710 (+ 364us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:44.862297 (+269587us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:44.862343 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:44.862404 (+ 61us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:44.862449 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.735364 (+1872915us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54000b4ad4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.735443 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:46.735710 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":619449088}
I20250831 13:58:46.753784 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.755656 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.756868 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.757946 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:52966 (request call id 444) took 1899 ms. Trace:
W20250831 13:58:46.701893 20495 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1270) took 2836 ms (client timeout 3292 ms). Trace:
I20250831 13:58:46.758471 20499 rpcz_store.cc:276] 0831 13:58:44.860397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.883337 (+ 22940us) service_pool.cc:225] Handling call
0831 13:58:46.757212 (+1873875us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54001626d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97c6e5 00007fb9bc9749ba 00007fb9c18d4079 00007fb9baf64644 00007fb9baf645d2 00007fb9baf64501 00007fb9baf6447c 00007fb9baf63732 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:46.757663 (+ 451us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":434106624}
W20250831 13:58:46.759028 20495 rpcz_store.cc:269] 0831 13:58:43.797794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.044447 (+246653us) service_pool.cc:225] Handling call
0831 13:58:46.633963 (+2589516us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b54000122d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:46.634335 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":32319360}
I20250831 13:58:46.761535 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1280) took 2269 ms. Trace:
I20250831 13:58:46.762290 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:46.752535 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:46.696161 20493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 820) took 1490 ms (client timeout 1563 ms). Trace:
I20250831 13:58:46.749691 22100 raft_consensus.cc:3053] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:46.762530 20574 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:46.765977 22100 raft_consensus.cc:738] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:58:46.765852 20493 rpcz_store.cc:269] 0831 13:58:45.206094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.253799 (+ 47705us) service_pool.cc:225] Handling call
0831 13:58:46.695686 (+1441887us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54001289d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:46.695979 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":726848384}
I20250831 13:58:46.766847 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 836) took 1453 ms. Trace:
I20250831 13:58:46.767297 20502 rpcz_store.cc:276] 0831 13:58:45.313920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.689932 (+1376012us) service_pool.cc:225] Handling call
0831 13:58:46.690534 (+ 602us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:46.720186 (+ 29652us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.766779 (+ 46593us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:46.748179 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:46.762138 20496 rpcz_store.cc:276] 0831 13:58:44.492008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.492125 (+ 117us) service_pool.cc:225] Handling call
0831 13:58:44.494763 (+ 2638us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:44.514814 (+ 20051us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:44.524372 (+ 9558us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:44.524913 (+ 541us) log.cc:844] Serialized 26 byte log entry
0831 13:58:46.303726 (+1778813us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:46.730039 (+426313us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b18003fd840. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9bc5c255c 00007fb9b99f777b 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171
0831 13:58:46.730426 (+ 387us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:46.730518 (+ 92us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.730653 (+ 135us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:46.739113 (+ 8460us) raft_consensus.cc:1683] finished
0831 13:58:46.739135 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:46.761324 (+ 22189us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":93583,"raft.queue_time_us":94130,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":28779,"spinlock_wait_cycles":42154496,"thread_start_us":148171,"threads_started":1,"wal-append.queue_time_us":396687}
I20250831 13:58:46.772074 20396 catalog_manager.cc:5582] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d reported cstate change: config changed from index 2 to 5, term changed from 5 to 7, ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:46.773649 22301 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:46.751044 20466 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:46.773723 20496 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:46.738296 20571 rpcz_store.cc:276] 0831 13:58:45.506587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.513226 (+ 6639us) service_pool.cc:225] Handling call
0831 13:58:46.699048 (+1185822us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b58000f0398. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:46.699401 (+ 353us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:46.720880 (+ 21479us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.726194 (+ 5314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3499776}
W20250831 13:58:46.775391 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 64529.6us
I20250831 13:58:46.768950 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 903) took 1064 ms. Trace:
I20250831 13:58:46.779476 20644 rpcz_store.cc:276] 0831 13:58:45.704433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.742285 (+1037852us) service_pool.cc:225] Handling call
0831 13:58:46.768899 (+ 26614us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:46.776304 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.776003 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1135) took 1081 ms. Trace:
I20250831 13:58:46.781474 20642 rpcz_store.cc:276] 0831 13:58:45.694532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.732166 (+1037634us) service_pool.cc:225] Handling call
0831 13:58:46.775897 (+ 43731us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9984}
W20250831 13:58:46.701344 20570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1270) took 2108 ms (client timeout 1743 ms). Trace:
I20250831 13:58:46.578554 22565 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:46.565465 22578 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 2.433s user 0.008s sys 0.011s
I20250831 13:58:46.569619 22589 raft_consensus.cc:3053] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:46.598044 22591 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:46.493310 22606 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.575901 22609 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:46.559847 22615 raft_consensus.cc:3053] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:46.931834 22646 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20250831 13:58:46.932168 22615 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:46.934095 22644 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:58:46.934777 22659 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:46.935829 22559 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:46.941066 20571 raft_consensus.cc:3053] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:46.940676 22664 consensus_queue.cc:1035] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:46.941442 20571 raft_consensus.cc:738] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:46.917409 22640 consensus_queue.cc:1046] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20250831 13:58:46.944821 22229 raft_consensus.cc:695] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:46.946498 22229 consensus_queue.cc:237] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.947598 22592 raft_consensus.cc:3053] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:46.948097 22592 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:46.949537 22592 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.950438 22606 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:46.953751 22429 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:46.938148 22674 consensus_queue.cc:1046] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20250831 13:58:46.533584 22614 consensus_queue.cc:237] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.955238 22674 raft_consensus.cc:3053] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:46.955705 22674 raft_consensus.cc:738] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:46.950480 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:46.957434 20574 raft_consensus.cc:2391] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 3: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:46.571301 22647 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.585282 22650 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:46.523643 22658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s user 0.047s sys 0.130s
I20250831 13:58:47.007205 20650 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 6 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 13:58:46.758522 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:47.009250 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 905) took 1219 ms. Trace:
I20250831 13:58:47.009667 20646 rpcz_store.cc:276] 0831 13:58:45.790037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.758213 (+968176us) service_pool.cc:225] Handling call
0831 13:58:47.009162 (+250949us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:46.934063 22615 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.011435 20646 raft_consensus.cc:1238] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:46.956700 22674 consensus_queue.cc:260] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.839489 20643 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:46.777557 22357 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:46.774618 22301 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:46.849843 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:46.846767 22538 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:46.495368 22431 consensus_queue.cc:237] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.852787 20495 raft_consensus.cc:1273] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:46.767397 22100 consensus_queue.cc:260] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.503854 22372 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:46.849310 22430 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:46.774868 20396 catalog_manager.cc:5582] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 1 to 2, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to ab7ade51fa134a1b814fab748f01d43d (127.18.233.132). New cstate: current_term: 2 leader_uuid: "ab7ade51fa134a1b814fab748f01d43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:58:46.914319 20647 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 7 LEADER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:46.917812 20644 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
W20250831 13:58:46.921769 20467 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.004s sys 0.016s
W20250831 13:58:46.923380 20575 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.023s sys 0.032s
I20250831 13:58:46.924454 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52966 (request call id 445) took 1303 ms. Trace:
W20250831 13:58:46.930763 20576 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.008s sys 0.006s
W20250831 13:58:47.027674 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.027911 20576 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.008s sys 0.007s
W20250831 13:58:46.931041 20617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.007s sys 0.007s
W20250831 13:58:47.029273 20617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s user 0.007s sys 0.008s
I20250831 13:58:46.958051 20571 consensus_queue.cc:260] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.030395 22372 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:47.031147 20649 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:47.031347 20571 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:47.032470 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1128) took 1834 ms. Trace:
I20250831 13:58:47.032871 20649 rpcz_store.cc:276] 0831 13:58:45.198185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.760173 (+1561988us) service_pool.cc:225] Handling call
0831 13:58:46.760625 (+ 452us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.031575 (+270950us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.032223 (+ 648us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54001ef354. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:47.032403 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":621491072}
I20250831 13:58:46.913865 22631 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:47.036351 22301 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:47.037108 22551 consensus_queue.cc:1046] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:47.037496 22100 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:46.926745 22589 raft_consensus.cc:738] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:46.998577 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 77861.9us
W20250831 13:58:46.979516 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 2.47685e+06us
I20250831 13:58:46.975643 22559 raft_consensus.cc:695] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:46.930701 21175 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:47.005280 22592 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:47.011521 22615 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:47.012173 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1132) took 1672 ms. Trace:
I20250831 13:58:46.930910 22626 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.175s
I20250831 13:58:47.013777 20645 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:47.012584 20572 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20250831 13:58:47.017498 22647 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:47.012292 20465 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.009s sys 0.021s
W20250831 13:58:47.020630 20643 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1134) took 1348 ms (client timeout 1696 ms). Trace:
I20250831 13:58:47.021412 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1219) took 1108 ms. Trace:
W20250831 13:58:47.025487 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44193.5us
I20250831 13:58:47.027386 20496 rpcz_store.cc:276] 0831 13:58:45.621477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.844212 (+1222735us) service_pool.cc:225] Handling call
0831 13:58:46.924320 (+ 80108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.014869 22650 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:47.040081 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46464 (request call id 58) took 1741 ms. Trace:
I20250831 13:58:47.074219 20647 rpcz_store.cc:276] 0831 13:58:45.285744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.842119 (+1556375us) service_pool.cc:225] Handling call
0831 13:58:46.884498 (+ 42379us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.026770 (+142272us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.026974 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.039688 20500 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20250831 13:58:47.041055 20467 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.005s sys 0.016s
I20250831 13:58:47.041422 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1129) took 1805 ms. Trace:
I20250831 13:58:47.078572 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1300) took 1151 ms. Trace:
I20250831 13:58:47.078475 20644 rpcz_store.cc:276] 0831 13:58:45.233588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.847950 (+1614362us) service_pool.cc:225] Handling call
0831 13:58:46.886716 (+ 38766us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.038623 (+151907us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.038821 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:47.043990 20575 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s user 0.023s sys 0.033s
I20250831 13:58:47.045508 20646 rpcz_store.cc:276] 0831 13:58:45.339761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.010482 (+1670721us) service_pool.cc:225] Handling call
0831 13:58:47.011021 (+ 539us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.011864 (+ 843us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.012105 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.079226 20500 rpcz_store.cc:276] 0831 13:58:45.927645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.698900 (+771255us) service_pool.cc:225] Handling call
0831 13:58:46.699368 (+ 468us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.078230 (+378862us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.078508 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.046249 20649 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
W20250831 13:58:47.049428 20643 rpcz_store.cc:269] 0831 13:58:45.670502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.670430 (+999928us) service_pool.cc:225] Handling call
0831 13:58:47.018949 (+348519us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.052090 22589 consensus_queue.cc:260] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:47.054371 20465 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.676s user 0.011s sys 0.022s
W20250831 13:58:47.057276 20645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 902) took 1353 ms (client timeout 1620 ms). Trace:
W20250831 13:58:47.059108 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.059702 22559 consensus_queue.cc:237] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.061268 22674 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:47.061638 22591 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:47.063355 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:47.089994 20572 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0c14599239fd4c99b3331692a513a092"
dest_uuid: "212b66976f874aa682cae0fb77928786"
mode: GRACEFUL
new_leader_uuid: "ab7ade51fa134a1b814fab748f01d43d"
from {username='slave'} at 127.0.0.1:49676
I20250831 13:58:47.091830 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 900) took 1563 ms. Trace:
I20250831 13:58:47.092196 20646 rpcz_store.cc:276] 0831 13:58:45.528064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.086528 (+1558464us) service_pool.cc:225] Handling call
0831 13:58:47.091196 (+ 4668us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.091518 (+ 322us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:47.091543 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:47.091586 (+ 43us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:47.091612 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.091665 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.091758 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21760}
I20250831 13:58:47.094107 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 906) took 1203 ms. Trace:
W20250831 13:58:47.083818 20645 rpcz_store.cc:269] 0831 13:58:45.704082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.483936 (+779854us) service_pool.cc:225] Handling call
0831 13:58:47.057014 (+573078us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b54001ee954. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:47.057202 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1215389568}
I20250831 13:58:47.066179 20466 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:47.067819 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 837) took 1753 ms. Trace:
I20250831 13:58:47.085681 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 896) took 1625 ms. Trace:
I20250831 13:58:47.014135 22606 consensus_queue.cc:1046] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.055s
I20250831 13:58:47.048434 22551 raft_consensus.cc:3053] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
W20250831 13:58:46.785913 20570 rpcz_store.cc:269] 0831 13:58:44.593014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.593142 (+ 128us) service_pool.cc:225] Handling call
0831 13:58:46.700804 (+2107662us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b540031e7d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:46.701084 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":562782976}
I20250831 13:58:47.094434 20646 rpcz_store.cc:276] 0831 13:58:45.890231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.093173 (+1202942us) service_pool.cc:225] Handling call
0831 13:58:47.093529 (+ 356us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.093783 (+ 254us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:47.093812 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:47.093864 (+ 52us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:47.093903 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.093967 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.094047 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.095953 20649 rpcz_store.cc:276] 0831 13:58:45.458450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.033935 (+1575485us) service_pool.cc:225] Handling call
0831 13:58:47.034459 (+ 524us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.083663 (+ 49204us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.083922 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.095657 20572 raft_consensus.cc:604] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Received request to transfer leadership to TS ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:47.049536 20573 rpcz_store.cc:276] 0831 13:58:45.911131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.660636 (+749505us) service_pool.cc:225] Handling call
0831 13:58:46.714896 (+ 54260us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:46.715357 (+ 461us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:46.715396 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:46.716206 (+ 810us) log.cc:844] Serialized 26 byte log entry
0831 13:58:46.886177 (+169971us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:46.898101 (+ 11924us) log.cc:844] Serialized 14 byte log entry
0831 13:58:46.912366 (+ 14265us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.912439 (+ 73us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:47.017559 (+105120us) raft_consensus.cc:1683] finished
0831 13:58:47.017588 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.019602 (+ 2014us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":11742,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":75,"raft.queue_time_us":15235,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":16420,"thread_start_us":169507,"threads_started":1,"wal-append.queue_time_us":298814}
I20250831 13:58:47.103334 20572 raft_consensus.cc:612] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250831 13:58:47.105087 20387 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 90a0a56228d04660b3cb351485be6593 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 58aded50ab0d4609a053647eb11b1f48 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:47.107601 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1292) took 1354 ms. Trace:
I20250831 13:58:47.107777 20649 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20250831 13:58:47.107991 20575 rpcz_store.cc:276] 0831 13:58:45.753600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.251920 (+498320us) service_pool.cc:225] Handling call
0831 13:58:46.252209 (+ 289us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:46.252579 (+ 370us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:46.252618 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:46.253171 (+ 553us) log.cc:844] Serialized 26 byte log entry
0831 13:58:47.107190 (+854019us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:47.107241 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.107319 (+ 78us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:47.107370 (+ 51us) raft_consensus.cc:1683] finished
0831 13:58:47.107391 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.107534 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":808,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":14586,"thread_start_us":759668,"threads_started":1,"wal-append.queue_time_us":622831}
I20250831 13:58:47.109381 20649 raft_consensus.cc:1273] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:47.109673 22551 raft_consensus.cc:738] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:47.109997 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:47.110659 22551 consensus_queue.cc:260] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.111447 20577 raft_consensus.cc:1238] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 13:58:47.112246 22551 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:47.115892 22605 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.116418 22605 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.117695 20498 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:47.095551 20497 rpcz_store.cc:276] 0831 13:58:45.314240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.692359 (+1378119us) service_pool.cc:225] Handling call
0831 13:58:46.739375 (+ 47016us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:46.776558 (+ 37183us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:46.776589 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:46.776638 (+ 49us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:46.776703 (+ 65us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.776764 (+ 61us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.067649 (+290885us) spinlock_profiling.cc:243] Waited 46.5 ms on lock 0x7b540013d650. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:47.067752 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":452,"raft.queue_time_us":64634,"raft.run_cpu_time_us":443,"raft.run_wall_time_us":44982,"spinlock_wait_cycles":106951552}
I20250831 13:58:47.122272 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 897) took 1663 ms. Trace:
I20250831 13:58:47.122713 20647 rpcz_store.cc:276] 0831 13:58:45.458836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.084331 (+1625495us) service_pool.cc:225] Handling call
0831 13:58:47.095057 (+ 10726us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.121377 (+ 26320us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:47.121403 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:47.122017 (+ 614us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:47.122057 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.122121 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.122223 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.123411 20571 raft_consensus.cc:1273] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:47.123939 20649 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250831 13:58:47.131525 20572 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
W20250831 13:58:47.139652 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 322841us
W20250831 13:58:47.149674 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.162158 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:47.163738 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 907) took 1181 ms. Trace:
I20250831 13:58:47.165192 20646 rpcz_store.cc:276] 0831 13:58:45.982545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.120112 (+1137567us) service_pool.cc:225] Handling call
0831 13:58:47.163030 (+ 42918us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.163401 (+ 371us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:47.163430 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:47.163497 (+ 67us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:47.163530 (+ 33us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.163583 (+ 53us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.163678 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.166703 20646 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20250831 13:58:47.152690 20645 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:47.167965 22429 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.169049 22429 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:47.150442 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 84429.9us
W20250831 13:58:47.164899 22697 log.cc:927] Time spent T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20250831 13:58:47.146646 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1117) took 2287 ms. Trace:
I20250831 13:58:47.168102 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1139) took 1224 ms. Trace:
I20250831 13:58:47.172391 20645 rpcz_store.cc:276] 0831 13:58:45.943687 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.119164 (+1175477us) service_pool.cc:225] Handling call
0831 13:58:47.152401 (+ 33237us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.167882 (+ 15481us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.168035 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.159026 22342 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.150607 22631 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.179399 22342 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.179924 22631 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.181835 22342 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:47.184741 22674 consensus_queue.cc:1046] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:58:47.186355 22681 raft_consensus.cc:3053] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:47.186765 22681 raft_consensus.cc:738] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:47.187816 22681 consensus_queue.cc:260] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:47.188524 21176 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.188964 22681 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:47.171139 20648 rpcz_store.cc:276] 0831 13:58:44.832297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.863471 (+ 31174us) service_pool.cc:225] Handling call
0831 13:58:44.863736 (+ 265us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:44.863920 (+ 184us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 13:58:44.884652 (+ 20732us) log.cc:844] Serialized 14 byte log entry
0831 13:58:46.684081 (+1799429us) spinlock_profiling.cc:243] Waited 25.5 ms on lock 0x7b1800471fa0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba2da 00007fb9bc9809da
0831 13:58:47.023906 (+339825us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:47.040864 (+ 16958us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:47.075828 (+ 34964us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.086626 (+ 10798us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.119576 (+ 32950us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":125563,"prepare.queue_time_us":20528,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":3480,"raft.queue_time_us":58890,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":12473,"spinlock_wait_cycles":58622336,"thread_start_us":227594,"threads_started":1,"wal-append.queue_time_us":2234105}
W20250831 13:58:47.154488 20617 leader_election.cc:336] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Timed out: RequestConsensusVote RPC to 127.18.233.132:35173 timed out after 1.986s (SENT)
I20250831 13:58:47.191478 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:47.182448 22631 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.149253 22606 raft_consensus.cc:3053] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:47.150924 22612 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:47.217298 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:47.200063 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:47.221971 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:47.228178 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.235397 20494 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:47.208202 22606 raft_consensus.cc:738] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:47.242619 20571 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:47.195879 22659 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20250831 13:58:47.245362 20466 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:47.238934 21205 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:47.259370 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1320) took 1114 ms. Trace:
I20250831 13:58:47.259704 20502 rpcz_store.cc:276] 0831 13:58:46.145112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.768416 (+623304us) service_pool.cc:225] Handling call
0831 13:58:46.843425 (+ 75009us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:46.894719 (+ 51294us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:47.026044 (+131325us) log.cc:844] Serialized 14 byte log entry
0831 13:58:47.259048 (+233004us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:47.259099 (+ 51us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:47.259142 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.259207 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.259312 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4363,"prepare.queue_time_us":113132,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":43,"thread_start_us":268493,"threads_started":2}
I20250831 13:58:47.266404 22713 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 13:58:47.266907 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 57947.7us
W20250831 13:58:47.267493 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 169458us
I20250831 13:58:47.267360 22713 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250831 13:58:47.268282 22713 consensus_queue.cc:1230] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20250831 13:58:47.269457 20466 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:47.269644 22713 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 13:58:47.270885 22548 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.265465 22612 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 13:58:47.277792 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.278474 22100 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:47.280244 22577 consensus_queue.cc:1046] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:47.266494 22614 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:47.258206 22429 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.294330 22531 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 13:58:47.294507 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 140086us
I20250831 13:58:47.295792 22695 consensus_queue.cc:1046] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:47.297195 22695 raft_consensus.cc:3053] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:47.297372 22431 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:47.297655 22695 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:47.297857 22694 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20250831 13:58:47.298050 22431 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.302727 22431 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:47.304262 22631 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:47.304922 22691 consensus_queue.cc:1046] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20250831 13:58:47.306802 22614 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.309211 22614 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.310022 22606 consensus_queue.cc:260] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:47.306814 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1311) took 1337 ms. Trace:
I20250831 13:58:47.311694 20494 rpcz_store.cc:276] 0831 13:58:45.969966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.745315 (+775349us) service_pool.cc:225] Handling call
0831 13:58:46.745758 (+ 443us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.235860 (+490102us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.306561 (+ 70701us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b540035b1d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:47.306753 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1125657856}
W20250831 13:58:47.312556 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.295800 20617 leader_election.cc:336] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173): Timed out: RequestConsensusVote RPC to 127.18.233.132:35173 timed out after 2.142s (SENT)
I20250831 13:58:47.315778 20617 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:47.281275 22100 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.019s
I20250831 13:58:47.315733 22631 raft_consensus.cc:695] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:47.327545 22631 consensus_queue.cc:237] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.318928 22359 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20250831 13:58:47.330216 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:47.330940 20493 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:47.331621 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:47.333498 20543 connection.cc:232] RPC call timeout handler was delayed by -2.5761878700000125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26880000000000004s for a response
W20250831 13:58:47.334215 20543 leader_election.cc:336] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.694s (SENT)
I20250831 13:58:47.334584 22685 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.335183 22685 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:47.336283 22699 log.cc:927] Time spent T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.276s user 0.002s sys 0.000s
I20250831 13:58:47.337208 22685 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:47.337462 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.338474 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:47.339191 20498 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1274) took 2991 ms (client timeout 1743 ms). Trace:
I20250831 13:58:47.339473 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:47.339569 20498 rpcz_store.cc:269] 0831 13:58:44.347436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:44.347908 (+ 472us) service_pool.cc:225] Handling call
0831 13:58:47.338948 (+2991040us) spinlock_profiling.cc:243] Waited 2.77 s on lock 0x7b540013ea54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:47.339111 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6367686528}
I20250831 13:58:47.340798 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1188) took 2541 ms. Trace:
W20250831 13:58:47.341948 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.343252 22716 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20250831 13:58:47.344255 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.345404 22605 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.346287 20651 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 5 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
W20250831 13:58:47.347527 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.358695 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.323654 22691 raft_consensus.cc:3053] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:47.323873 22541 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:47.351192 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 892) took 2098 ms. Trace:
I20250831 13:58:47.360977 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:47.353307 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.322829 22606 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:47.347013 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.340804 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.355190 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1130) took 2109 ms. Trace:
I20250831 13:58:47.350147 20493 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:47.378358 20642 rpcz_store.cc:276] 0831 13:58:45.253294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.350317 (+2097023us) service_pool.cc:225] Handling call
0831 13:58:47.350696 (+ 379us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.350882 (+ 186us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:47.350911 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:47.350965 (+ 54us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:47.350990 (+ 25us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.351058 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.351149 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.378901 20651 rpcz_store.cc:276] 0831 13:58:45.245581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.345778 (+2100197us) service_pool.cc:225] Handling call
0831 13:58:47.346086 (+ 308us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.354896 (+ 8810us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.355128 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.316175 22577 consensus_queue.cc:1230] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20250831 13:58:47.426826 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.341207 20569 rpcz_store.cc:276] 0831 13:58:44.799383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:45.232079 (+432696us) service_pool.cc:225] Handling call
0831 13:58:45.241458 (+ 9379us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:45.245020 (+ 3562us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:45.245064 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:45.245651 (+ 587us) log.cc:844] Serialized 26 byte log entry
0831 13:58:46.702540 (+1456889us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b18003b8360. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171 00007fb9bc94909f
0831 13:58:46.907578 (+205038us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:46.907632 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:46.907757 (+ 125us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:47.322920 (+415163us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b48002601d8. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c1838c41 00007fb9c182fde7 00007fb9c182fc7a 00007fb9c192506f 00007fb9c1924d37 00007fb9c193935c 00007fb9c1939308 00007fb9c1939288 00007fb9c1939243 00007fb9c19384ec 00007fb9c08b6656
0831 13:58:47.335547 (+ 12627us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:47.340484 (+ 4937us) raft_consensus.cc:1683] finished
0831 13:58:47.340518 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.340639 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":94,"raft.run_cpu_time_us":12765,"raft.run_wall_time_us":2076191,"spinlock_wait_cycles":63830400,"thread_start_us":204655,"threads_started":1,"wal-append.queue_time_us":1814347}
I20250831 13:58:47.361169 22691 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:47.439004 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:47.430341 20495 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:58:47.437206 20642 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 5 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
W20250831 13:58:47.449617 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.450286 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.450666 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.451030 20543 connection.cc:232] RPC call timeout handler was delayed by -2.1127825689999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30150000000000005s for a response
W20250831 13:58:47.451475 20543 connection.cc:232] RPC call timeout handler was delayed by -1.8679522579999457s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19720000000000001s for a response
W20250831 13:58:47.451893 20543 connection.cc:232] RPC call timeout handler was delayed by -1.3122869439999931s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15860000000000002s for a response
W20250831 13:58:47.452306 20543 connection.cc:232] RPC call timeout handler was delayed by -1.2816869439999437s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16200000000000003s for a response
W20250831 13:58:47.452715 20543 connection.cc:232] RPC call timeout handler was delayed by -1.1871869439999614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17250000000000001s for a response
W20250831 13:58:47.453116 20543 connection.cc:232] RPC call timeout handler was delayed by -1.1520869439999615s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
W20250831 13:58:47.453527 20543 connection.cc:232] RPC call timeout handler was delayed by -1.1232869439999718s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20250831 13:58:47.453907 20543 connection.cc:232] RPC call timeout handler was delayed by -1.0350869439999428s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20250831 13:58:47.454279 20543 connection.cc:232] RPC call timeout handler was delayed by -1.0269869439999866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19030000000000002s for a response
I20250831 13:58:47.382176 22541 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:47.361397 22429 consensus_queue.cc:1046] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.088s
I20250831 13:58:47.472174 20574 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:47.474018 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.431432 22685 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.473603 20498 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:58:47.477500 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.429975 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 159160us
I20250831 13:58:47.482762 20575 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:47.486349 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.486996 20646 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:47.487025 22660 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.483659 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:47.487742 22660 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.488835 20642 raft_consensus.cc:1273] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:47.487991 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.490381 20646 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:47.492610 22722 consensus_queue.cc:1035] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:47.495007 22660 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:47.495182 22722 consensus_queue.cc:1035] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:47.497130 20651 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:47.498757 22722 consensus_queue.cc:1035] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250831 13:58:47.498939 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 203214us
I20250831 13:58:47.500351 20617 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:47.501149 22359 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:47.501789 22359 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.502859 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 899) took 2028 ms. Trace:
I20250831 13:58:47.503721 20568 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 8. Current term is 9. Ops: []
I20250831 13:58:47.503842 22359 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:47.504009 22342 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:47.504534 22342 raft_consensus.cc:695] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:47.505373 22342 consensus_queue.cc:237] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.478456 22609 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.501474 22691 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:47.513260 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.503211 20644 rpcz_store.cc:276] 0831 13:58:45.474874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.079692 (+1604818us) service_pool.cc:225] Handling call
0831 13:58:47.091171 (+ 11479us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:47.133679 (+ 42508us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:47.133725 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:47.221110 (+ 87385us) log.cc:844] Serialized 52 byte log entry
0831 13:58:47.312247 (+ 91137us) raft_consensus.cc:1643] Marking committed up to 1
0831 13:58:47.312273 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.312319 (+ 46us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:47.502639 (+190320us) raft_consensus.cc:1683] finished
0831 13:58:47.502660 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.502806 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38463,"raft.queue_time_us":42438,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":15882,"spinlock_wait_cycles":17152,"thread_start_us":58723,"threads_started":1,"wal-append.queue_time_us":183668}
I20250831 13:58:47.515859 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:47.517375 21205 raft_consensus.cc:3053] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:47.517872 21205 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:47.518988 21205 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:47.520525 21205 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:47.521787 22589 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:47.527566 20493 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20250831 13:58:47.535681 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 51402us
W20250831 13:58:47.541986 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.484925 22493 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 13:58:47.608687 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.616797 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.623819 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.486111 22685 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.654136 20466 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:47.476877 22695 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:47.661078 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.662602 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.669015 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:47.620914 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 76986.4us
W20250831 13:58:47.696632 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 153615us
I20250831 13:58:47.546211 22609 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:47.692203 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.656692 22691 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:47.661566 22614 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 13:58:47.705003 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 46443.6us
I20250831 13:58:47.706245 20466 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:47.701258 21175 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.696326 22548 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.642608 22577 consensus_queue.cc:1035] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.064s
I20250831 13:58:47.701449 22229 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.707332 22614 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:47.703667 22551 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.721935 22586 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.722517 22586 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:47.724270 22613 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.724810 22613 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:47.725160 22586 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:47.728466 22554 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.713717 22615 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:47.730448 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 188580us
I20250831 13:58:47.712684 22681 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.767872 22548 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.708374 22429 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.019s
I20250831 13:58:47.717841 22631 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.659874 22612 consensus_queue.cc:1046] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20250831 13:58:47.715615 22614 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:47.729672 22613 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:47.781742 20651 raft_consensus.cc:3053] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:47.769057 22615 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:47.773475 22695 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:47.719112 21708 consensus_queue.cc:1046] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20250831 13:58:47.719689 22229 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.802119 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.803141 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 891) took 2562 ms. Trace:
W20250831 13:58:47.804925 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.729105 22551 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:47.784027 22528 consensus_queue.cc:1046] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20250831 13:58:47.829782 20651 raft_consensus.cc:738] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:47.838348 20650 rpcz_store.cc:276] 0831 13:58:45.230459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.775274 (+1544815us) service_pool.cc:225] Handling call
0831 13:58:46.775801 (+ 527us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:47.007720 (+231919us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.792769 (+785049us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b540028bc54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:47.793011 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":522938752}
W20250831 13:58:47.841424 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 111779us
I20250831 13:58:47.790288 22721 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.188s
I20250831 13:58:47.788021 22554 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.823352 22723 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:47.841820 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 71727us
I20250831 13:58:47.807909 22681 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:47.848287 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 124085us
I20250831 13:58:47.848531 20644 raft_consensus.cc:2408] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 13:58:47.832233 22615 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:47.840731 21205 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20250831 13:58:47.852195 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.852658 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.843858 22612 consensus_queue.cc:1230] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 13:58:47.840227 22685 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.850858 20543 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:47.850208 20651 consensus_queue.cc:260] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.859737 20650 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:47.854363 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:47.832464 22631 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.855576 22528 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:47.870175 20651 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:47.863399 22717 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20250831 13:58:47.875890 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.879122 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.881273 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.882156 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:47.868067 22609 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.885723 20497 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20250831 13:58:47.888862 20497 pending_rounds.cc:85] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 2
I20250831 13:58:47.889184 20497 pending_rounds.cc:107] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250831 13:58:47.889462 20497 raft_consensus.cc:2887] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:47.886524 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1304) took 1945 ms. Trace:
I20250831 13:58:47.889055 22565 raft_consensus.cc:3053] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:47.881060 22100 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:47.893853 22357 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:47.894280 22357 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:47.895192 22548 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.892606 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:47.900214 22705 log.cc:927] Time spent T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.841s user 0.007s sys 0.010s
I20250831 13:58:47.900395 22358 consensus_queue.cc:1046] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20250831 13:58:47.900887 22327 raft_consensus.cc:3053] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:47.901265 22327 raft_consensus.cc:738] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:47.901342 22358 raft_consensus.cc:3053] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:47.901711 22358 raft_consensus.cc:738] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:47.878746 22733 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250831 13:58:47.902230 22327 consensus_queue.cc:260] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.906279 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:47.906909 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:47.909452 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:47.865574 22723 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:47.890121 20501 rpcz_store.cc:276] 0831 13:58:45.936374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.701306 (+764932us) service_pool.cc:225] Handling call
0831 13:58:46.701911 (+ 605us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:46.995509 (+293598us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:46.995545 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:47.027869 (+ 32324us) log.cc:844] Serialized 26 byte log entry
0831 13:58:47.119709 (+ 91840us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:47.119784 (+ 75us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.876913 (+757129us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54002dfd54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:47.876963 (+ 50us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:47.877016 (+ 53us) raft_consensus.cc:1683] finished
0831 13:58:47.877030 (+ 14us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.881268 (+ 4238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43871,"raft.queue_time_us":56121,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":24841,"spinlock_wait_cycles":527653632,"thread_start_us":59254,"threads_started":1,"wal-append.queue_time_us":159636}
I20250831 13:58:47.912410 20617 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:47.914582 22686 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.853s user 0.000s sys 0.011s
I20250831 13:58:47.891878 22565 raft_consensus.cc:738] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:47.926440 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:47.919431 20646 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:47.935313 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.926455 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.921507 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.936079 20502 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
W20250831 13:58:47.936901 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 90773.8us
W20250831 13:58:47.939656 22731 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20250831 13:58:47.939771 22643 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:58:47.940197 22685 consensus_queue.cc:1046] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:47.935110 20569 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:47.903517 22327 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:47.951433 20617 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:47.920060 22338 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20250831 13:58:47.955157 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.955801 20498 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:47.956650 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:47.957345 20498 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:47.958082 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:47.959249 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.959391 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1297) took 2202 ms. Trace:
I20250831 13:58:47.959591 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:47.961035 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.959800 20576 rpcz_store.cc:276] 0831 13:58:45.756986 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:46.303222 (+546236us) service_pool.cc:225] Handling call
0831 13:58:46.304000 (+ 778us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:46.304357 (+ 357us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:46.304388 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:46.305094 (+ 706us) log.cc:844] Serialized 26 byte log entry
0831 13:58:47.028564 (+723470us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:47.028940 (+ 376us) log.cc:844] Serialized 14 byte log entry
0831 13:58:47.029180 (+ 240us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.029288 (+ 108us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:47.913938 (+884650us) spinlock_profiling.cc:243] Waited 7.73 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:47.959113 (+ 45175us) raft_consensus.cc:1683] finished
0831 13:58:47.959196 (+ 83us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.959337 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":842,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":43,"raft.queue_time_us":15495,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":9998,"spinlock_wait_cycles":17784064,"thread_start_us":655812,"threads_started":1,"wal-append.queue_time_us":741408}
I20250831 13:58:47.960389 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.966210 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.925731 22357 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.969844 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:47.969997 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:47.971887 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.973068 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:47.966987 20502 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:47.974310 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:47.974515 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:47.976365 22613 raft_consensus.cc:3053] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:47.976935 22613 raft_consensus.cc:738] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:47.978883 22554 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.980644 20543 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:58:47.981369 22730 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20250831 13:58:47.981812 22100 raft_consensus.cc:2762] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250831 13:58:47.983024 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.983713 20648 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:47.985294 22681 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:47.973141 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 901) took 2422 ms. Trace:
I20250831 13:58:47.988826 20643 rpcz_store.cc:276] 0831 13:58:45.550665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.972285 (+2421620us) service_pool.cc:225] Handling call
0831 13:58:47.972611 (+ 326us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:47.972838 (+ 227us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:47.972861 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:47.972903 (+ 42us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:47.972951 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:47.973001 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:47.973092 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:47.970623 20496 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:58:47.985440 22729 log.cc:927] Time spent T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.132s user 0.003s sys 0.000s
I20250831 13:58:47.889809 22528 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:47.905958 22532 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20250831 13:58:47.958395 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:47.995317 20575 consensus_meta.cc:287] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.509s user 0.008s sys 0.002s
W20250831 13:58:48.001602 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 50280.5us
I20250831 13:58:47.986357 22229 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:48.006666 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:48.024093 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.011641 20651 raft_consensus.cc:1273] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:48.013083 20648 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:48.014607 20466 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:47.982491 20543 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.018293 20617 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:48.027657 20573 raft_consensus.cc:1238] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:48.017524 20575 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:48.022966 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:47.983549 22631 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.032666 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.036285 20650 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 5 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:48.037318 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.038452 20617 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:48.040217 20643 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:48.048200 20466 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:47.978137 22613 consensus_queue.cc:260] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:48.050951 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.050516 20569 raft_consensus.cc:1273] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:48.037046 22100 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:48.037866 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:48.045122 22726 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.334s user 0.003s sys 0.000s
W20250831 13:58:48.059613 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 101364us
I20250831 13:58:48.037477 22359 raft_consensus.cc:2802] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:48.067025 20501 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
W20250831 13:58:48.043191 22606 consensus_meta.cc:287] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.660s user 0.010s sys 0.000s
I20250831 13:58:48.073608 20493 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:47.902663 22358 consensus_queue.cc:260] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.024065 22687 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:48.042120 22554 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:48.087059 22723 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:48.063457 20396 catalog_manager.cc:5582] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:48.042971 22551 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:48.087047 22554 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:48.088457 22554 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:48.090216 22728 log.cc:927] Time spent T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.363s user 0.002s sys 0.000s
I20250831 13:58:48.062384 22631 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:48.090879 22631 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:48.091262 22631 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:48.098239 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 115921us
I20250831 13:58:48.099057 22429 raft_consensus.cc:3053] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:48.099584 22429 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:48.099428 20543 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.100584 22687 raft_consensus.cc:695] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:48.042191 22548 consensus_queue.cc:1035] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20250831 13:58:48.101841 22687 consensus_queue.cc:237] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.104331 22459 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:48.083910 20498 raft_consensus.cc:1273] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:48.079874 22674 consensus_queue.cc:1046] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20250831 13:58:48.085443 22369 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:48.085821 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:48.114188 21206 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:48.115312 22727 log.cc:927] Time spent T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.412s user 0.003s sys 0.000s
I20250831 13:58:48.116052 22674 consensus_queue.cc:1230] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 13:58:48.112915 22369 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:48.116878 22369 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:48.116990 22100 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:48.118227 22381 raft_consensus.cc:3053] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:48.118547 20543 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.118755 22381 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:48.119470 22551 consensus_queue.cc:1046] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:48.120312 22551 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:48.119894 22381 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.120883 22551 raft_consensus.cc:3053] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:48.121276 22551 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:48.121527 22381 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:48.122035 22551 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.123557 22551 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:48.106555 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:48.126827 20467 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.066763 22649 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20250831 13:58:48.085980 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:48.128089 22674 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250831 13:58:48.069638 22586 consensus_queue.cc:1162] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "44d163094add49049cc749547d891602" responder_term: 5 status { last_received { term: 4 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 13:58:48.128893 21205 raft_consensus.cc:2408] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 13:58:48.067340 22357 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.130590 20467 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.131357 22586 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:48.132164 22429 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.132531 20467 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.133227 22586 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:48.133311 22429 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:48.133510 20468 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:48.134496 22586 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:48.134900 22586 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:48.134835 20468 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:48.135478 22586 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:48.136090 22357 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:48.108564 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:48.138772 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 66483.2us
I20250831 13:58:48.139580 20467 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:48.067260 22359 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:48.125387 20543 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.141870 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:48.151015 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.144798 22359 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:48.162469 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.171828 21708 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.147051 22691 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:48.171290 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:48.187124 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 52812.3us
I20250831 13:58:48.179702 21708 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.189848 20543 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.084390 22565 consensus_queue.cc:260] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.204943 20575 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250831 13:58:48.208504 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.210532 22182 raft_consensus.cc:2762] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250831 13:58:48.211341 20569 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:48.216518 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.218885 20649 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:48.226992 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.227772 20571 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:48.185850 22372 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.229069 22372 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:48.231396 22372 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:48.192251 22542 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:48.232851 22542 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:48.233269 22542 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:48.234313 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:48.235649 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.237224 22614 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.242215 22565 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:48.243072 22528 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:48.248513 22589 consensus_meta.cc:287] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.726s user 0.008s sys 0.005s
I20250831 13:58:48.249608 20494 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:48.255376 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 97432.7us
I20250831 13:58:48.256831 20617 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:48.257136 22554 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.260529 22586 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:48.261054 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:48.261756 22631 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.263757 22643 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:48.264075 22369 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.264954 20502 raft_consensus.cc:2408] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 13:58:48.265501 22615 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.258244 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.269770 20496 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 9.
I20250831 13:58:48.314495 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:48.315510 20496 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:48.275033 20573 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:48.192860 22358 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:48.319521 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.277697 22691 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 13:58:48.277910 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 71475.7us
I20250831 13:58:48.292610 22554 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:48.279501 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:48.280033 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 96599.8us
I20250831 13:58:48.281069 22430 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:48.281692 22695 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:48.281692 22532 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250831 13:58:48.282774 21708 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.284140 20500 raft_consensus.cc:1273] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:48.286232 22614 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.335620 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:48.336635 20496 raft_consensus.cc:2391] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:48.287089 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:48.284174 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:48.291828 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.306638 20645 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:48.340373 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.308543 20494 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:48.306217 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:48.276387 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:48.325551 22541 consensus_meta.cc:287] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.852s user 0.004s sys 0.008s
I20250831 13:58:48.324798 20498 raft_consensus.cc:1238] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:48.278642 22660 consensus_queue.cc:1046] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 13:58:48.328835 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.342133 20648 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:48.325361 20493 raft_consensus.cc:2391] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:48.345464 20499 raft_consensus.cc:2391] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:48.349597 22369 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:48.357036 20498 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:48.356967 22359 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.358109 20643 raft_consensus.cc:1238] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:48.358376 20498 raft_consensus.cc:1273] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:48.360556 22586 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:48.362352 22613 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:48.362847 22541 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.365192 22541 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:48.366945 22660 consensus_queue.cc:1230] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Peer 212b66976f874aa682cae0fb77928786 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 13:58:48.367440 22660 raft_consensus.cc:3053] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:58:48.367775 22660 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:48.368613 22660 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.370997 22735 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.375432 20574 raft_consensus.cc:1273] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:48.376024 20573 raft_consensus.cc:2408] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 13:58:48.373394 20497 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
W20250831 13:58:48.377069 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.325906 22429 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.371483 22735 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:48.390462 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 106351us
I20250831 13:58:48.391546 20467 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.384991 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:58:48.386222 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:48.387328 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.390604 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1334) took 1052 ms. Trace:
I20250831 13:58:48.393846 20496 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:48.394111 20494 rpcz_store.cc:276] 0831 13:58:47.337999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:47.338110 (+ 111us) service_pool.cc:225] Handling call
0831 13:58:48.390421 (+1052311us) spinlock_profiling.cc:243] Waited 910 ms on lock 0x7b540035bbd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:48.390532 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2094079872}
I20250831 13:58:48.384079 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:48.396831 20500 raft_consensus.cc:2391] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:48.376649 22554 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:48.406471 20648 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:58:48.409133 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.413050 20573 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:48.414382 20466 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:48.400209 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.344856 22695 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.427232 20499 raft_consensus.cc:1273] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20250831 13:58:48.432585 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.433166 20574 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:48.437701 20616 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:48.438573 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.433748 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.398777 22589 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:48.460515 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.460959 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.430578 22614 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
W20250831 13:58:48.465517 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 118769us
I20250831 13:58:48.470237 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.465641 20572 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:48.472319 20577 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:48.413565 22565 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:48.398855 22381 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:48.411013 22429 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.445748 20500 raft_consensus.cc:1238] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
W20250831 13:58:48.477080 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.482522 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:48.480604 22614 raft_consensus.cc:695] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:48.483404 20497 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:48.451850 22691 consensus_queue.cc:1046] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20250831 13:58:48.477733 22643 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:48.479199 22565 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:48.493170 20577 raft_consensus.cc:1273] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:48.508366 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:48.509925 20642 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:48.478277 22338 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.464845 22577 consensus_queue.cc:1046] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20250831 13:58:48.490428 22551 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.469964 21175 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.465574 22182 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20250831 13:58:48.534047 20572 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 6. Ops: []
I20250831 13:58:48.500542 22691 consensus_queue.cc:1230] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 13:58:48.512156 22565 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:48.496285 22631 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.515512 22359 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:48.494325 22381 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.545310 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:48.551280 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 61222.7us
I20250831 13:58:48.551918 20500 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 3.
I20250831 13:58:48.549116 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:48.553123 20494 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:48.553324 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.527561 22643 raft_consensus.cc:695] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:48.529718 22615 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.538187 22577 raft_consensus.cc:3053] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:48.557941 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.554438 20572 raft_consensus.cc:2408] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 13:58:48.571940 20577 raft_consensus.cc:2408] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 13:58:48.573727 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.574906 20573 raft_consensus.cc:2408] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250831 13:58:48.575587 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.578110 20543 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.530144 22551 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.581674 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.582769 20574 raft_consensus.cc:2408] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:58:48.530100 22338 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:48.555733 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 93067us
W20250831 13:58:48.590253 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 118780us
I20250831 13:58:48.567096 22577 raft_consensus.cc:738] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:48.595096 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:48.573845 22735 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.599617 20648 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: []
W20250831 13:58:48.599066 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.607758 20494 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:48.614625 20500 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:48.599428 22695 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.592105 22691 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:48.580207 22614 consensus_queue.cc:237] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:48.660169 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 82145us
I20250831 13:58:48.633412 22735 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:48.665442 20543 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:48.684902 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:48.665911 22748 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.192s user 0.000s sys 0.001s
I20250831 13:58:48.689450 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:48.690186 20497 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:48.693390 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.694130 20574 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 3.
I20250831 13:58:48.686422 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:48.689589 20387 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 90a0a56228d04660b3cb351485be6593 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 58aded50ab0d4609a053647eb11b1f48 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:48.695438 20573 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:48.687502 20575 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:48.698110 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.703534 20572 raft_consensus.cc:1273] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250831 13:58:48.666478 22429 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.704780 20397 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59942 (request call id 147) took 10679 ms. Trace:
I20250831 13:58:48.711689 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:48.712127 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.715035 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.716251 20467 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:48.716601 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:48.716997 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.717589 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:48.718065 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:48.718359 20573 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:48.718609 20577 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 8
I20250831 13:58:48.718845 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.724381 20500 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:48.724300 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:48.725068 20575 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:48.726572 20467 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.726585 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:48.728205 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:48.728533 20570 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:48.728890 20572 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:48.731765 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:48.733418 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 68096.4us
I20250831 13:58:48.645319 22643 consensus_queue.cc:237] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.733444 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:48.742594 20645 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:48.724289 22586 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.728207 22695 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:48.726076 22615 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:48.758744 20467 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:48.739573 22745 log.cc:927] Time spent T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.204s user 0.000s sys 0.001s
I20250831 13:58:48.669822 22577 consensus_queue.cc:260] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.727865 22381 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:48.743784 22753 log.cc:927] Time spent T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.170s user 0.000s sys 0.001s
I20250831 13:58:48.765630 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.757678 22631 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:48.772478 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:48.767179 22542 consensus_meta.cc:287] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.533s user 0.013s sys 0.000s
I20250831 13:58:48.762475 22615 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:48.787739 22615 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:48.789192 22755 log.cc:927] Time spent T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250831 13:58:48.733971 21175 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:48.793429 20642 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:48.777235 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.779543 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:48.780241 22338 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:48.780741 22691 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:48.781896 22347 raft_consensus.cc:3053] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:48.762094 22605 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.016s
I20250831 13:58:48.787289 20645 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 6 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 13:58:48.766042 22551 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.773387 22381 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:48.762212 22586 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.807653 20575 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:48.795871 22577 raft_consensus.cc:3058] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:48.802414 20577 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:58:48.797641 22347 raft_consensus.cc:738] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:48.712692 20397 rpcz_store.cc:276] 0831 13:58:38.025466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:38.025619 (+ 153us) service_pool.cc:225] Handling call
0831 13:58:48.704531 (+10678912us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:41.357658 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:41.615975 (+258317us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:41.618585 (+ 2610us) write_op.cc:435] Acquired schema lock
0831 13:58:41.620487 (+ 1902us) tablet.cc:660] Decoding operations
0831 13:58:41.754116 (+133629us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:41.754215 (+ 99us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:41.754263 (+ 48us) tablet.cc:683] Acquiring locks for 10 operations
0831 13:58:41.755380 (+ 1117us) tablet.cc:699] Row locks acquired
0831 13:58:41.755402 (+ 22us) write_op.cc:260] PREPARE: finished
0831 13:58:41.755593 (+ 191us) write_op.cc:270] Start()
0831 13:58:41.755753 (+ 160us) write_op.cc:276] Timestamp: P: 1756648721755533 usec, L: 0
0831 13:58:41.755777 (+ 24us) op_driver.cc:348] REPLICATION: starting
0831 13:58:41.756585 (+ 808us) log.cc:844] Serialized 4059 byte log entry
0831 13:58:41.813418 (+ 56833us) op_driver.cc:464] REPLICATION: finished
0831 13:58:41.816038 (+ 2620us) write_op.cc:301] APPLY: starting
0831 13:58:41.816117 (+ 79us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:41.816289 (+ 172us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:41.816380 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:41.817331 (+ 951us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:41.817492 (+ 161us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0831 13:58:41.817520 (+ 28us) write_op.cc:312] APPLY: finished
0831 13:58:41.817763 (+ 243us) log.cc:844] Serialized 76 byte log entry
0831 13:58:41.817923 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:41.818223 (+ 300us) write_op.cc:454] Released schema lock
0831 13:58:41.818275 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:41.818620 (+ 345us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:41.818649 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":87495,"reported_tablets":37,"spinlock_wait_cycles":32128,"child_traces":[["op",{"apply.queue_time_us":2374,"apply.run_cpu_time_us":2353,"apply.run_wall_time_us":2531,"num_ops":10,"prepare.queue_time_us":213396,"prepare.run_cpu_time_us":12627,"prepare.run_wall_time_us":474502,"raft.queue_time_us":720,"raft.run_cpu_time_us":2595,"raft.run_wall_time_us":2979,"replication_time_us":57469,"thread_start_us":85363,"threads_started":4,"wal-append.queue_time_us":54386,"wal-append.run_cpu_time_us":2232,"wal-append.run_wall_time_us":1013666}]]}
I20250831 13:58:48.805141 22605 consensus_queue.cc:1035] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:58:48.805889 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.791432 22695 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:48.801366 20497 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
W20250831 13:58:48.815039 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 361600us
I20250831 13:58:48.797796 22528 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:58:48.800474 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 82241.8us
I20250831 13:58:48.791826 22542 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:48.828155 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:48.826720 22381 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:58:48.832086 22760 log.cc:927] Time spent T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20250831 13:58:48.827409 22695 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.843050 22691 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:48.843420 22759 log.cc:927] Time spent T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20250831 13:58:48.843634 22691 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.846037 22691 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.849455 20570 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:48.851794 20573 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:48.811707 22551 consensus_queue.cc:1046] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
W20250831 13:58:48.857045 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 86545.9us
I20250831 13:58:48.806844 22758 consensus_queue.cc:1046] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.140s
I20250831 13:58:48.858589 20499 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:48.852706 22551 consensus_queue.cc:1230] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 13:58:48.828266 22761 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20250831 13:58:48.881351 20498 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.014s sys 0.001s
W20250831 13:58:48.881914 20498 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s user 0.014s sys 0.001s
I20250831 13:58:48.882858 20466 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:48.843081 22695 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.899773 20466 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:48.900696 22381 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:48.872068 22761 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:48.908723 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 314374us
I20250831 13:58:48.901021 22528 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
W20250831 13:58:48.915968 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 99040.2us
I20250831 13:58:48.916893 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:48.917784 20616 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.919709 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.913204 22182 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.010s
I20250831 13:58:48.923751 22182 raft_consensus.cc:2762] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250831 13:58:48.923933 22532 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:48.924335 22182 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:48.924763 22182 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.927891 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.928552 20647 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 3.
I20250831 13:58:48.931042 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.932090 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:48.932193 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.932655 20645 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:48.933386 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:48.923967 22542 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.901156 22381 raft_consensus.cc:695] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:48.937059 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.937765 20647 raft_consensus.cc:2391] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:48.937506 22381 consensus_queue.cc:237] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:48.938975 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 52908.9us
I20250831 13:58:48.940371 20543 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:48.941887 20543 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.905632 22565 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:48.944437 22565 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.944818 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:48.945098 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:48.931244 22764 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20250831 13:58:48.944967 22565 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:48.926420 22735 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:48.950038 20572 raft_consensus.cc:2408] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20250831 13:58:48.952191 22772 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:48.948264 20645 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:48.954667 20617 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:48.907411 22761 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:58:48.972323 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 166500us
I20250831 13:58:48.975478 20467 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.976631 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:48.977771 22381 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:48.977893 20466 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:48.981086 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 54743.5us
I20250831 13:58:48.950955 22586 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:48.976656 20467 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:48.926635 22182 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:49.005271 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 50714.8us
I20250831 13:58:49.005249 20467 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.008237 20467 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:49.010558 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 90981.2us
I20250831 13:58:49.010979 22614 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:49.008000 22782 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:49.012795 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:48.969151 22625 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20250831 13:58:49.008672 22430 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:48.991719 22615 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.023290 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:49.002084 22528 raft_consensus.cc:695] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:49.009258 22586 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:49.040562 20617 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:49.013218 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.040064 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.034777 22782 raft_consensus.cc:695] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:49.071697 22780 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:49.072402 22577 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:49.072798 21203 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:49.090183 20644 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:49.033353 22182 raft_consensus.cc:2762] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250831 13:58:49.034600 22372 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.034521 22614 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:49.023419 22695 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.022863 22615 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:49.071251 22744 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.430s user 0.000s sys 0.002s
W20250831 13:58:49.076309 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 63179.7us
I20250831 13:58:49.116174 20467 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:49.119825 22756 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.479s user 0.000s sys 0.003s
I20250831 13:58:49.121973 22614 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:49.124068 22782 consensus_queue.cc:237] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.127301 22660 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:49.068794 22649 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250831 13:58:49.101368 22342 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20250831 13:58:49.147203 22605 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.147833 22605 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.101351 22358 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:49.101783 22592 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:49.085783 22338 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.155202 22338 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.155509 22347 consensus_queue.cc:260] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.094743 22532 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250831 13:58:49.108384 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:49.073772 22643 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
W20250831 13:58:49.159500 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:49.163460 20647 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250831 13:58:49.108810 22780 consensus_queue.cc:1046] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:58:49.108963 22528 consensus_queue.cc:237] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.164920 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.165544 20572 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:49.110221 22372 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.167893 20645 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:49.092571 22577 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.090646 22430 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20250831 13:58:49.179881 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:49.182675 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 64103.1us
I20250831 13:58:49.187593 20466 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:49.183836 20468 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:49.188599 20575 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250831 13:58:49.185034 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:49.192880 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:49.193435 20574 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:49.188956 20575 pending_rounds.cc:85] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 3
I20250831 13:58:49.194623 20575 pending_rounds.cc:107] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250831 13:58:49.194959 20575 raft_consensus.cc:2887] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 13:58:49.195138 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 97713.9us
I20250831 13:58:49.087390 22586 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:49.204730 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.205340 20568 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:49.206772 20467 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:49.207620 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.211506 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:49.216071 20572 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:49.217777 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:49.220083 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.220908 20577 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:49.208407 20575 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:49.186342 22430 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:58:49.236689 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 106320us
I20250831 13:58:49.194419 22589 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:49.200532 22429 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:49.232882 20575 raft_consensus.cc:1273] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:49.198443 22347 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:49.235502 20467 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.239938 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.240527 20569 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:49.231937 20577 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:49.247279 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:49.217325 22615 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:49.250918 20617 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:49.259872 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.260658 20647 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:49.262390 20541 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:49.265710 22551 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:49.266088 22551 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:49.266608 22551 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:49.270820 22543 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.059s
I20250831 13:58:49.195760 22613 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.043s
I20250831 13:58:49.290473 20572 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:49.238485 22430 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:49.239089 22429 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:49.293298 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.300184 20569 raft_consensus.cc:2408] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 7 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:49.251716 22786 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 13:58:49.298837 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 50487.4us
I20250831 13:58:49.273342 22577 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.258509 22589 raft_consensus.cc:695] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:49.256117 22615 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:49.315572 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 64774.9us
I20250831 13:58:49.320307 20616 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:49.320529 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.332427 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.306715 22429 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:49.262846 22586 consensus_queue.cc:1035] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20250831 13:58:49.334213 20543 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:49.299861 22372 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:49.333071 20569 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:49.312321 22605 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.292733 22649 consensus_queue.cc:1046] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.039s
I20250831 13:58:49.315361 22713 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:49.329574 22338 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:49.331995 22358 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20250831 13:58:49.337218 20466 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.323580 22615 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:49.329829 22541 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:49.419394 20543 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:49.354051 22780 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:49.426641 20650 raft_consensus.cc:1238] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:49.357569 22605 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:49.358032 22541 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:49.430840 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 111504us
I20250831 13:58:49.418318 22381 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.436936 22787 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.169s
W20250831 13:58:49.446445 22792 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20250831 13:58:49.426573 22722 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:49.447499 22722 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:49.447974 22722 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:49.452572 20574 raft_consensus.cc:2466] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:49.454250 20467 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:49.459383 20568 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:49.459393 20573 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 13:58:49.436417 20617 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:49.472462 20568 raft_consensus.cc:1273] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250831 13:58:49.358284 22713 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:49.421954 22644 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:58:49.434535 22541 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:49.479188 20617 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:49.490751 20571 raft_consensus.cc:1273] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20250831 13:58:49.487469 20617 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:49.481804 22358 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:49.491223 20571 pending_rounds.cc:85] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 2
I20250831 13:58:49.497443 20571 pending_rounds.cc:107] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250831 13:58:49.497771 20571 raft_consensus.cc:2887] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 13:58:49.498617 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 141541us
I20250831 13:58:49.437039 22381 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.425292 22589 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:49.501564 20569 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:49.528729 22691 raft_consensus.cc:3053] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:49.529174 22691 raft_consensus.cc:738] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:49.530015 22691 consensus_queue.cc:260] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.531173 22691 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:58:49.500212 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 180003us
I20250831 13:58:49.533047 20616 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:49.485159 22773 consensus_queue.cc:1046] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20250831 13:58:49.489672 22790 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:49.535290 20645 raft_consensus.cc:1273] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20250831 13:58:49.535761 22790 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.536880 22338 raft_consensus.cc:3053] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:49.537395 22338 raft_consensus.cc:738] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:49.537820 22790 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.484774 22614 raft_consensus.cc:2802] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:49.499522 22229 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:58:49.436600 22761 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:49.538456 22338 consensus_queue.cc:260] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.540875 22338 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:49.541193 22761 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.542832 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:49.543431 20645 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:49.546810 22778 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.547559 22787 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:49.548076 22787 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.495740 22605 consensus_queue.cc:1035] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.054s
I20250831 13:58:49.549777 22787 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.503257 22359 raft_consensus.cc:2802] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:49.551452 22713 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.552671 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.553418 22229 raft_consensus.cc:695] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:49.554186 20543 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:49.554451 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:49.535326 22773 raft_consensus.cc:3053] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:49.554373 22229 consensus_queue.cc:237] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.555050 22773 raft_consensus.cc:738] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:49.555930 22773 consensus_queue.cc:260] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.557045 22773 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:49.555105 20499 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:49.483326 22644 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:49.563016 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:49.574620 20648 raft_consensus.cc:1273] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:58:49.576859 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.583346 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.557552 22359 raft_consensus.cc:695] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:49.561172 22614 raft_consensus.cc:695] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:49.587109 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 66827.2us
I20250831 13:58:49.571985 22644 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:58:49.597855 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:49.575891 22347 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:49.600809 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.601644 20645 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:49.601899 20651 raft_consensus.cc:2408] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:49.579064 22369 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:49.603644 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:49.605418 20569 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:49.576035 22358 raft_consensus.cc:695] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:49.609608 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 110403us
I20250831 13:58:49.610476 20543 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:49.563921 22695 consensus_queue.cc:1035] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20250831 13:58:49.615056 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.616007 20577 raft_consensus.cc:2391] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:49.616827 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.587190 22100 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.607290 22369 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:49.619292 22787 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.637889 20467 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.621949 22767 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:49.647090 20467 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.598896 22605 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20250831 13:58:49.591773 22778 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.625543 22631 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.651347 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:49.655160 20494 raft_consensus.cc:1273] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250831 13:58:49.655742 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 45397us
I20250831 13:58:49.659942 20543 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:49.657848 20647 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:49.636752 22347 raft_consensus.cc:2762] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250831 13:58:49.653354 22787 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.691443 22381 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.691640 22787 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.632558 22100 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.693805 22100 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:49.665735 21205 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s user 0.003s sys 0.007s
I20250831 13:58:49.620237 21175 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:49.694768 21205 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.610s user 0.003s sys 0.008s
W20250831 13:58:49.695475 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 92226.8us
W20250831 13:58:49.660861 22660 consensus_meta.cc:287] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.531s user 0.010s sys 0.000s
I20250831 13:58:49.696879 22713 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:49.695739 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47742 (request call id 91) took 1384 ms. Trace:
W20250831 13:58:49.671691 22796 log.cc:927] Time spent T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20250831 13:58:49.633975 22359 consensus_queue.cc:237] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.673000 22801 consensus_queue.cc:1046] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 13:58:49.676272 22795 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.253s user 0.001s sys 0.000s
I20250831 13:58:49.650431 22767 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:49.680662 22631 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.659057 20501 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:49.752242 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:49.683132 20573 raft_consensus.cc:1273] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:58:49.686861 22614 consensus_queue.cc:237] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.688071 22358 consensus_queue.cc:237] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.688793 22615 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.644694 22369 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:49.699249 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:49.637851 22551 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.755931 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:49.700474 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:49.715895 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 68870.6us
I20250831 13:58:49.710412 20569 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:49.724638 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.723213 22782 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:49.679339 22778 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:49.743484 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 83618.6us
I20250831 13:58:49.725530 22542 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:49.749493 22541 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:49.700440 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.646749 22543 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.097s
I20250831 13:58:49.756055 20494 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:49.649971 22649 consensus_queue.cc:1046] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.098s
W20250831 13:58:49.769560 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 131917us
I20250831 13:58:49.756791 20649 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:49.697983 21205 rpcz_store.cc:276] 0831 13:58:48.329929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:48.346794 (+ 16865us) service_pool.cc:225] Handling call
0831 13:58:48.434795 (+ 88001us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:48.979949 (+545154us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:49.003844 (+ 23895us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:49.019550 (+ 15706us) log.cc:844] Serialized 26 byte log entry
0831 13:58:49.082262 (+ 62712us) raft_consensus.cc:1643] Marking committed up to 7
0831 13:58:49.695188 (+612926us) log.cc:844] Serialized 14 byte log entry
0831 13:58:49.695403 (+ 215us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:49.695477 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:49.695525 (+ 48us) raft_consensus.cc:1683] finished
0831 13:58:49.695536 (+ 11us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:49.695658 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2344,"prepare.queue_time_us":567255,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":67706,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":25353,"spinlock_wait_cycles":1180416,"thread_start_us":1129806,"threads_started":3,"wal-append.queue_time_us":47460}
I20250831 13:58:49.745733 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.746995 22780 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:49.755156 22767 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:49.770309 20495 raft_consensus.cc:1238] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250831 13:58:49.783565 20543 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:49.774245 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.784762 20569 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:49.785126 20543 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:49.745275 22430 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.785403 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.782570 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:49.791949 22430 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:49.792797 20650 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:49.794834 22778 consensus_queue.cc:1035] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:49.799854 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.800155 22615 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.800454 20571 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:49.801432 22631 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.801642 20571 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20250831 13:58:49.803457 22803 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20250831 13:58:49.805589 22541 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:49.809170 20575 raft_consensus.cc:1273] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:58:49.815207 20543 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:49.815613 20574 raft_consensus.cc:1273] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:49.817224 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:49.831959 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:49.838747 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 56223.9us
I20250831 13:58:49.766178 22722 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.848066 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.850212 20569 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:49.851401 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:49.788659 22229 raft_consensus.cc:3053] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:49.848003 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:49.814757 22347 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:49.852849 20467 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:49.770622 22542 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.862144 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:49.828176 22780 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:49.870632 20466 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.840951 22778 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:58:49.781560 21175 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:49.843164 22758 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:49.852276 22327 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:49.892043 20466 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.853626 22615 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.840394 22554 consensus_queue.cc:1035] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
I20250831 13:58:49.854573 22229 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:49.900099 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 60977.9us
W20250831 13:58:49.873420 22429 consensus_meta.cc:287] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.526s user 0.010s sys 0.000s
I20250831 13:58:49.875646 22778 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:49.883813 22528 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:49.883241 22695 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:49.892350 22327 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:49.920053 20466 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:49.928306 20543 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:49.890436 22577 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:49.937062 20499 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 4.
I20250831 13:58:49.944532 20496 raft_consensus.cc:1238] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 13:58:49.944880 20543 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:49.898486 22786 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 13:58:49.949754 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 50276.3us
I20250831 13:58:49.901297 22713 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Leader election won for term 4
W20250831 13:58:49.899312 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 131228us
I20250831 13:58:49.928890 22778 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:58:49.957423 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 52075.1us
I20250831 13:58:49.928740 22528 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:49.958894 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:49.915280 22541 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:49.938256 22327 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:49.938337 22359 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:49.966508 20617 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:49.979115 22812 log.cc:927] Time spent T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.156s user 0.000s sys 0.000s
I20250831 13:58:49.964653 22713 raft_consensus.cc:695] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:49.955621 22758 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:49.957422 22780 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:49.954417 22649 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20250831 13:58:49.958959 22551 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:49.914968 22772 consensus_queue.cc:1046] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20250831 13:58:49.963148 22528 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:49.938306 22429 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.929697 22808 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:49.947268 22660 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:49.905750 22615 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.929255 22695 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.950167 22577 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:49.985111 22631 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:49.988641 22782 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:49.983121 22229 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.017074 20501 raft_consensus.cc:2391] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 8: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:50.018811 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:49.986447 22430 consensus_queue.cc:1035] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20250831 13:58:50.027482 20650 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:49.986711 22541 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.029922 20541 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:50.030951 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.031682 20499 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:50.034420 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.049103 22782 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.048116 22713 consensus_queue.cc:237] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:50.051806 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:49.990303 22359 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:50.024242 22758 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.014655 22613 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:50.087782 22810 log.cc:927] Time spent T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.417s user 0.003s sys 0.000s
I20250831 13:58:50.089782 22347 raft_consensus.cc:3053] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:50.089499 22767 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.090128 22347 raft_consensus.cc:738] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:50.090971 22347 consensus_queue.cc:260] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.091825 22819 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:50.092290 22819 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.094211 22819 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.094414 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.095011 20495 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:50.095103 22804 raft_consensus.cc:3053] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:50.095463 22804 raft_consensus.cc:738] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:50.096262 22804 consensus_queue.cc:260] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.097379 22804 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:50.036312 22543 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:50.010167 22660 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.035073 22605 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:50.040576 22722 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.107056 20495 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
W20250831 13:58:50.076416 22826 log.cc:927] Time spent T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250831 13:58:50.057590 22772 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:50.092098 22347 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:50.082448 22229 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:50.082659 22542 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.119434 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.040254 22625 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20250831 13:58:50.106139 22780 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20250831 13:58:50.131493 20499 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:50.127238 22542 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:50.122517 22772 raft_consensus.cc:695] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:50.138160 22772 consensus_queue.cc:237] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.139389 22541 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:50.140551 22429 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.141022 22359 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:50.098969 22644 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.137084 22577 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.098816 22613 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:50.145553 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:50.153283 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 72162.2us
I20250831 13:58:50.168874 20543 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:50.169519 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:50.170214 20573 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 8
W20250831 13:58:50.174126 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 156585us
W20250831 13:58:50.177332 22823 log.cc:927] Time spent T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
W20250831 13:58:50.177992 22827 log.cc:927] Time spent T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20250831 13:58:50.179872 22577 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:58:50.183990 22577 raft_consensus.cc:695] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 9 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:50.100960 22758 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.185981 20543 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.112296 22605 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:50.179975 22824 log.cc:927] Time spent T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
W20250831 13:58:50.189043 20387 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 90a0a56228d04660b3cb351485be6593 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 58aded50ab0d4609a053647eb11b1f48 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:50.144037 22369 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:50.187072 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 157975us
W20250831 13:58:50.197639 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 277674us
I20250831 13:58:50.226657 20543 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:50.187227 20647 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:50.230773 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.137101 22542 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.235129 20617 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:50.179987 22660 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:50.180997 22767 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:50.242983 20617 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:50.246186 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.247044 20569 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:58:50.250869 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:50.253069 20467 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:50.255081 20495 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:50.217087 22722 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:50.169039 22691 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
I20250831 13:58:50.259931 20651 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:50.265574 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.271940 20573 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:58:50.273530 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.274713 20467 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:50.276154 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:50.276845 20573 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:58:50.281214 20495 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250831 13:58:50.284420 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.255677 22359 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:50.291319 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:50.291610 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.256026 22615 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.292395 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.259541 22631 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:50.302030 20577 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:50.302901 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:50.303602 20577 raft_consensus.cc:2391] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 8: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:50.254719 22761 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:50.304023 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:50.262953 22551 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250831 13:58:50.278417 22695 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.264302 22767 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:50.255131 22338 consensus_queue.cc:1046] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20250831 13:58:50.310832 20648 raft_consensus.cc:2391] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:50.312352 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.317593 20571 raft_consensus.cc:1273] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20250831 13:58:50.318089 20571 pending_rounds.cc:85] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 2
I20250831 13:58:50.318348 20571 pending_rounds.cc:107] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20250831 13:58:50.318616 20571 raft_consensus.cc:2887] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 13:58:50.334559 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 108013us
I20250831 13:58:50.336510 20577 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:50.340611 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.240581 22577 consensus_queue.cc:237] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.344746 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:50.311013 22687 consensus_queue.cc:1046] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20250831 13:58:50.346195 22644 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:50.346225 22551 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:50.347149 22767 raft_consensus.cc:695] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:50.347545 22687 consensus_queue.cc:1230] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [LEADER]: Peer 58aded50ab0d4609a053647eb11b1f48 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 13:58:50.347688 22369 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:50.348126 22767 consensus_queue.cc:237] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.307027 22631 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:50.311656 22782 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.315681 22808 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.364656 20573 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
W20250831 13:58:50.369856 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:50.334039 22359 raft_consensus.cc:695] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:50.357040 22814 log.cc:927] Time spent T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.442s user 0.002s sys 0.000s
W20250831 13:58:50.340052 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 60391.1us
I20250831 13:58:50.357226 22631 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:50.316426 22761 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:50.393633 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 119044us
W20250831 13:58:50.405337 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 160482us
I20250831 13:58:50.380736 22643 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.410043 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.363695 22839 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
W20250831 13:58:50.405644 22829 log.cc:927] Time spent T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.220s user 0.001s sys 0.000s
I20250831 13:58:50.422556 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.423002 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.424937 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.425812 20646 raft_consensus.cc:3053] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:50.426187 20646 raft_consensus.cc:738] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:50.429519 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.423445 20466 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.429430 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:50.432466 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 72734.9us
I20250831 13:58:50.371486 22808 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.436666 20577 raft_consensus.cc:1273] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:58:50.450377 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:50.437517 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.437587 20466 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.438457 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.436386 22758 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.439603 22613 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:50.441107 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.442030 22589 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:50.442765 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:50.443303 22761 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:50.451835 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:50.451015 20577 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 9
W20250831 13:58:50.466408 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:50.467433 20568 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 6
I20250831 13:58:50.469178 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.466941 20466 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.426654 22839 raft_consensus.cc:695] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:50.408147 22528 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.478257 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.482491 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.468340 20646 consensus_queue.cc:260] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.417848 22643 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.478436 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.486128 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:50.487542 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.488725 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.489655 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.448621 22542 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.497752 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.479104 22808 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.482749 22613 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:50.477717 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:50.506584 20495 raft_consensus.cc:2366] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:58:50.509253 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.466718 22853 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:50.515645 20648 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:50.516419 20648 raft_consensus.cc:604] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:50.516806 20648 raft_consensus.cc:612] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:58:50.519711 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.520721 20496 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:50.523212 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.522146 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.523772 20498 raft_consensus.cc:2374] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6. Re-sending same reply.
I20250831 13:58:50.531805 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.532768 20499 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:50.534217 20568 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:50.536618 22850 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20250831 13:58:50.538069 22836 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250831 13:58:50.542795 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.544430 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.504474 22847 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:50.545931 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.478813 22327 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.548877 22327 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.508165 21174 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:50.556443 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 59449us
I20250831 13:58:50.559906 20576 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:50.561508 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:50.561467 20467 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.562207 20645 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 8
I20250831 13:58:50.565898 20577 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:58:50.567301 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:50.579656 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.508388 22643 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:50.475180 22359 consensus_queue.cc:237] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:50.507931 22839 consensus_queue.cc:237] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.583834 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.584937 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.585879 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.525731 22528 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.546169 22847 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:50.527845 21205 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:50.599468 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.655481 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.655766 22861 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:50.656389 20502 raft_consensus.cc:2374] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5. Re-sending same reply.
I20250831 13:58:50.657153 20645 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:58:50.657320 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.658012 20502 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:58:50.658593 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 60822.8us
I20250831 13:58:50.659433 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:50.660866 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 56293.5us
I20250831 13:58:50.659389 20467 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.663415 20541 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:50.663875 20617 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:50.665300 20645 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:50.670882 20502 raft_consensus.cc:2366] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 8.
I20250831 13:58:50.675329 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.675669 20467 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.681655 20498 raft_consensus.cc:2391] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:50.689393 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.683425 20617 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:50.689975 20502 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
W20250831 13:58:50.680891 22811 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20250831 13:58:50.679375 22369 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:50.692358 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.680677 22589 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:50.696398 20498 raft_consensus.cc:2366] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:50.680938 22327 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:58:50.696069 22369 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:50.698431 22369 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:50.708315 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:50.690629 22862 log.cc:927] Time spent T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250831 13:58:50.682665 22853 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:50.690479 22852 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.716192 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.718508 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.719378 20502 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:50.720400 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.721225 20502 raft_consensus.cc:2366] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:50.722054 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.683279 22347 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20250831 13:58:50.724946 22369 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.727216 22369 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:50.691031 22860 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.730360 20502 raft_consensus.cc:2391] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:50.731904 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.733589 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:50.739773 20502 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:50.715755 22856 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.742448 20502 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a7956988b57844cfad1028024ac745e6"
dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
mode: GRACEFUL
new_leader_uuid: "44d163094add49049cc749547d891602"
from {username='slave'} at 127.0.0.1:35300
W20250831 13:58:50.733443 22778 consensus_meta.cc:287] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.748s user 0.005s sys 0.008s
I20250831 13:58:50.718161 22327 raft_consensus.cc:695] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:50.742115 22855 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:50.752230 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1202) took 1051 ms. Trace:
I20250831 13:58:50.752550 20647 rpcz_store.cc:276] 0831 13:58:49.698572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.700186 (+ 1614us) service_pool.cc:225] Handling call
0831 13:58:50.750184 (+1049998us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54002eb654. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:50.750314 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":964208384}
I20250831 13:58:50.731498 22852 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.756546 22852 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.757154 20498 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
W20250831 13:58:50.757827 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:50.758487 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:50.763712 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:50.778484 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 70984.1us
I20250831 13:58:50.745853 22778 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.782830 20573 pending_rounds.cc:85] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 5
I20250831 13:58:50.783129 20573 pending_rounds.cc:107] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20250831 13:58:50.783394 20573 raft_consensus.cc:2887] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:50.783776 20543 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.785162 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.776468 22852 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.787144 20543 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.740259 22860 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.789904 20493 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 9: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:50.744632 22856 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.791298 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:50.799700 20577 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:50.753885 22855 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.792485 20569 raft_consensus.cc:1273] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:50.807403 20467 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.821172 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:50.812234 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.822037 20650 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:50.788396 22347 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:50.834162 20502 raft_consensus.cc:604] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Received request to transfer leadership to TS 44d163094add49049cc749547d891602
I20250831 13:58:50.836902 20502 raft_consensus.cc:612] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250831 13:58:50.818238 22229 consensus_meta.cc:287] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.695s user 0.008s sys 0.002s
I20250831 13:58:50.834829 20467 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.822727 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:50.816020 22805 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 1.153s user 0.003s sys 0.000s
I20250831 13:58:50.842828 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1368) took 1190 ms. Trace:
I20250831 13:58:50.766891 22631 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.817205 22778 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.843675 20497 rpcz_store.cc:276] 0831 13:58:49.652166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.652276 (+ 110us) service_pool.cc:225] Handling call
0831 13:58:49.652631 (+ 355us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:49.652869 (+ 238us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 13:58:49.727392 (+ 74523us) log.cc:844] Serialized 14 byte log entry
0831 13:58:50.760092 (+1032700us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b1800432100. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba2da 00007fb9bc9809da
0831 13:58:50.842507 (+ 82415us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:50.842551 (+ 44us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:50.842579 (+ 28us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:50.842647 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:50.842772 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5083,"prepare.queue_time_us":74357,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":6479,"spinlock_wait_cycles":40490752,"thread_start_us":77025,"threads_started":1}
I20250831 13:58:50.833899 22758 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:50.802660 22872 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.847229 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:50.847748 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 60649.1us
I20250831 13:58:50.835911 22761 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:50.846824 20467 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:50.799911 22852 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.843290 22681 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:50.849675 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:50.850629 20543 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.851025 20496 raft_consensus.cc:2391] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:50.860062 22871 log.cc:927] Time spent T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20250831 13:58:50.852013 22875 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:50.814167 22847 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.847995 22839 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:50.861284 20497 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:50.854293 22856 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.855015 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:50.859936 22808 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:50.861410 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1373) took 1553 ms. Trace:
I20250831 13:58:50.851532 21176 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.862598 20572 rpcz_store.cc:276] 0831 13:58:49.307796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.421773 (+113977us) service_pool.cc:225] Handling call
0831 13:58:49.433165 (+ 11392us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:49.450938 (+ 17773us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:49.450967 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:49.451414 (+ 447us) log.cc:844] Serialized 26 byte log entry
0831 13:58:49.572522 (+121108us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:49.572564 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:49.572627 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:50.752042 (+1179415us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:50.856677 (+104635us) raft_consensus.cc:1683] finished
0831 13:58:50.856698 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:50.861351 (+ 4653us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15952,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":63,"spinlock_wait_cycles":61616896,"thread_start_us":120777,"threads_started":1,"wal-append.queue_time_us":167576}
I20250831 13:58:50.853197 22855 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.788535 22327 consensus_queue.cc:237] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.872321 20543 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.872891 22860 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.873577 20543 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.869202 22859 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:50.876382 22859 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:50.869841 22856 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:50.859086 22778 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20250831 13:58:50.887048 22859 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:50.869014 22839 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:50.897550 22839 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.899713 22839 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.905161 20499 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:58:50.907106 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:50.911271 20496 raft_consensus.cc:1238] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 13:58:50.912624 20496 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 13:58:50.913792 20496 raft_consensus.cc:1238] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20250831 13:58:50.917809 20497 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:58:50.906670 20499 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20250831 13:58:50.881764 22758 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:50.921651 20496 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20250831 13:58:50.926858 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1380) took 1152 ms. Trace:
W20250831 13:58:50.927201 22816 log.cc:927] Time spent T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Append to log took a long time: real 1.012s user 0.001s sys 0.003s
W20250831 13:58:50.934885 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:50.875471 22855 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:58:50.937925 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 64447.4us
I20250831 13:58:50.947928 20574 rpcz_store.cc:276] 0831 13:58:49.774552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.774723 (+ 171us) service_pool.cc:225] Handling call
0831 13:58:49.775006 (+ 283us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:49.816420 (+ 41414us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:50.926627 (+1110207us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b54003d4ad4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:50.926805 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92686592}
I20250831 13:58:50.949031 20496 pending_rounds.cc:85] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 3
I20250831 13:58:50.975617 20496 pending_rounds.cc:107] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20250831 13:58:50.976119 20496 raft_consensus.cc:2887] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:50.858976 22872 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:50.984544 20499 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20250831 13:58:50.989462 20543 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.990631 20499 raft_consensus.cc:1273] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:58:50.992777 20543 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:50.998026 20543 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:50.992065 22767 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.005s sys 0.018s
I20250831 13:58:50.988605 22852 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.993480 22758 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:51.004964 22551 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:51.006875 22874 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:51.021427 20543 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.014876 22631 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:50.995640 22780 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:58:51.001827 22767 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s user 0.005s sys 0.018s
I20250831 13:58:50.995625 22855 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:51.030001 22855 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:51.030035 22856 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:51.029332 20543 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.030774 22856 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.032123 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:51.032667 22855 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:51.032950 22856 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.034292 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1367) took 1382 ms. Trace:
W20250831 13:58:51.001740 22787 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.013s user 0.020s sys 0.013s
I20250831 13:58:51.034716 20493 rpcz_store.cc:276] 0831 13:58:49.651914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.652039 (+ 125us) service_pool.cc:225] Handling call
0831 13:58:50.273360 (+621321us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:51.034113 (+760753us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b5400186fd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:51.034235 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1241327616}
I20250831 13:58:51.034061 22856 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 13:58:51.039004 22787 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.055s user 0.021s sys 0.013s
W20250831 13:58:51.044312 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 197588us
I20250831 13:58:51.044636 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.043943 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:51.046469 21203 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:51.047441 22681 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:51.048794 20493 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:58:51.053123 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:51.054018 20574 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:51.057464 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.077013 20568 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:51.077543 20647 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250831 13:58:51.061116 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1370) took 1408 ms. Trace:
I20250831 13:58:51.078254 20500 rpcz_store.cc:276] 0831 13:58:49.652610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.652724 (+ 114us) service_pool.cc:225] Handling call
0831 13:58:49.652996 (+ 272us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:49.794933 (+141937us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:49.794974 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:49.795386 (+ 412us) log.cc:844] Serialized 26 byte log entry
0831 13:58:49.893124 (+ 97738us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:49.893161 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:51.060711 (+1167550us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400162454. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:51.060766 (+ 55us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:51.060815 (+ 49us) raft_consensus.cc:1683] finished
0831 13:58:51.060831 (+ 16us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:51.061052 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27524,"raft.queue_time_us":28592,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":75,"spinlock_wait_cycles":262284672,"thread_start_us":97322,"threads_started":1,"wal-append.queue_time_us":186853,"wal-append.run_cpu_time_us":1787,"wal-append.run_wall_time_us":1041962}
I20250831 13:58:51.066301 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.081874 20575 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:51.066495 20616 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.076376 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.057897 20543 leader_election.cc:400] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 8.
W20250831 13:58:51.068989 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:51.064244 20571 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250831 13:58:51.067315 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:51.067232 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 383869us
I20250831 13:58:51.024523 22758 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.089489 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:51.091837 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.102762 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.105408 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.067498 22644 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:58:51.133894 22644 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:51.134413 22644 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:51.136806 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.056265 22879 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:58:51.141314 22605 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.141821 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:51.110524 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.113427 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.113736 20500 raft_consensus.cc:1238] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250831 13:58:51.070119 22359 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:51.113588 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.147665 20570 raft_consensus.cc:2408] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:51.118607 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:51.148300 20569 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:51.073071 22541 raft_consensus.cc:3053] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
W20250831 13:58:51.120330 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 68628.3us
I20250831 13:58:51.151564 20543 leader_election.cc:400] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:51.152000 20543 leader_election.cc:403] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:51.153725 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:51.155598 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.156272 20643 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:51.158478 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:51.093833 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.162985 20649 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:51.122429 22882 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 13:58:51.123020 22815 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:51.124305 22887 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:51.125758 20467 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:51.074123 22872 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.130694 22822 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:51.068569 22845 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.656s user 0.000s sys 0.002s
I20250831 13:58:51.144572 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:51.120069 22229 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:58:51.179723 20500 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626"
dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
mode: GRACEFUL
new_leader_uuid: "44d163094add49049cc749547d891602"
from {username='slave'} at 127.0.0.1:35300
I20250831 13:58:51.180464 20500 raft_consensus.cc:604] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Received request to transfer leadership to TS 44d163094add49049cc749547d891602
I20250831 13:58:51.180891 20500 raft_consensus.cc:612] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20250831 13:58:51.171682 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 46005us
I20250831 13:58:51.173229 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.149924 22541 raft_consensus.cc:738] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:51.182904 20468 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:51.181977 20467 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:51.174274 20616 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.191826 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:51.163417 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 47090.4us
I20250831 13:58:51.195258 20543 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.172227 22772 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:51.168150 22815 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:51.184830 22839 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:51.171799 22543 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:51.161171 22804 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20250831 13:58:51.160615 22359 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:51.202005 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 57510us
I20250831 13:58:51.203784 20543 leader_election.cc:400] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:51.194977 22229 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:58:51.204319 22837 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:51.164808 22836 consensus_queue.cc:1046] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20250831 13:58:51.167665 22869 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20250831 13:58:51.205976 20543 leader_election.cc:403] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:51.206975 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.208623 20616 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.210069 20570 raft_consensus.cc:2391] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:51.192704 22614 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:51.190863 22872 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:51.212186 20500 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250831 13:58:51.202893 22577 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:51.203524 22839 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:51.217705 20648 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:51.183485 22831 consensus_queue.cc:1035] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20250831 13:58:51.221982 20569 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:51.202802 22815 raft_consensus.cc:3058] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:51.202250 22369 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:51.205168 22543 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:51.207511 22801 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:51.208700 22229 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:51.234850 20467 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:51.220811 22772 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:51.211447 22614 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:51.230487 22879 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:51.234202 22839 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:51.237818 20500 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:51.259042 22836 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:58:51.264355 22898 consensus_queue.cc:1035] T c719f2a4c3ea4418937337e8e3dd2164 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:51.264639 20642 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:58:51.232172 22873 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250831 13:58:51.265197 20642 raft_consensus.cc:604] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 1 LEADER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:58:51.266247 20570 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:58:51.266834 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20250831 13:58:51.266781 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 63096.2us
I20250831 13:58:51.187909 22822 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.255125 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1373) took 1598 ms. Trace:
I20250831 13:58:51.183934 22887 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.214372 22592 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20250831 13:58:51.268208 20543 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:51.295523 20501 rpcz_store.cc:276] 0831 13:58:49.656260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.751950 (+ 95690us) service_pool.cc:225] Handling call
0831 13:58:51.254891 (+1502941us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54002df354. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:51.255056 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":607221120}
I20250831 13:58:51.245186 22229 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
W20250831 13:58:51.300621 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 79087.8us
I20250831 13:58:51.250413 22860 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:51.260118 22614 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:51.310384 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 60774.4us
I20250831 13:58:51.310643 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.311576 20543 leader_election.cc:400] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:58:51.311913 20543 leader_election.cc:403] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:58:51.211373 22837 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.303397 22875 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:51.306296 22551 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:51.245839 22369 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:51.232785 22541 consensus_queue.cc:260] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.305455 22758 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:51.326407 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:51.316886 22890 log.cc:927] Time spent T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20250831 13:58:51.310232 22856 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:51.316738 22892 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20250831 13:58:51.301383 22852 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20250831 13:58:51.310174 22836 raft_consensus.cc:695] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:51.335457 22822 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:51.335920 22900 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:51.339893 22369 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:51.295724 22879 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.353680 22644 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:51.355159 22881 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:58:51.357530 21206 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20250831 13:58:51.359346 22888 log.cc:927] Time spent T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.127s user 0.001s sys 0.000s
W20250831 13:58:51.371712 22894 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250831 13:58:51.372099 22541 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:51.374825 22880 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
W20250831 13:58:51.375698 22893 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20250831 13:58:51.375864 22840 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.957s user 0.008s sys 0.003s
I20250831 13:58:51.314229 22860 raft_consensus.cc:695] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:51.378121 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 60707.2us
W20250831 13:58:51.375942 22804 consensus_queue.cc:1162] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "212b66976f874aa682cae0fb77928786" responder_term: 6 status { last_received { term: 6 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 4 } }
I20250831 13:58:51.346477 20396 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54880 (request call id 39) took 7582 ms. Trace:
I20250831 13:58:51.377615 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250831 13:58:51.384931 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:51.322716 22551 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:51.418305 22879 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:51.392912 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 85339.5us
I20250831 13:58:51.419056 22860 consensus_queue.cc:237] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.419917 20543 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:51.420641 22887 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.421008 20543 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.394326 20570 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:58:51.415933 20396 rpcz_store.cc:276] 0831 13:58:43.767806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:43.786727 (+ 18921us) service_pool.cc:225] Handling call
0831 13:58:51.346386 (+7559659us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:58:48.600015 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:58:48.600822 (+ 807us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:58:48.600889 (+ 67us) write_op.cc:435] Acquired schema lock
0831 13:58:48.600928 (+ 39us) tablet.cc:660] Decoding operations
0831 13:58:48.641184 (+ 40256us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:58:48.665715 (+ 24531us) write_op.cc:641] Partition lock acquired for write op
0831 13:58:48.669618 (+ 3903us) tablet.cc:683] Acquiring locks for 3 operations
0831 13:58:48.773457 (+103839us) tablet.cc:699] Row locks acquired
0831 13:58:48.775354 (+ 1897us) write_op.cc:260] PREPARE: finished
0831 13:58:48.797768 (+ 22414us) write_op.cc:270] Start()
0831 13:58:48.819731 (+ 21963us) write_op.cc:276] Timestamp: P: 1756648728792746 usec, L: 0
0831 13:58:48.827940 (+ 8209us) op_driver.cc:348] REPLICATION: starting
0831 13:58:48.913920 (+ 85980us) log.cc:844] Serialized 1438 byte log entry
0831 13:58:49.341124 (+427204us) op_driver.cc:464] REPLICATION: finished
0831 13:58:49.658646 (+317522us) write_op.cc:301] APPLY: starting
0831 13:58:49.727108 (+ 68462us) tablet.cc:1365] starting BulkCheckPresence
0831 13:58:49.749147 (+ 22039us) tablet.cc:1368] finished BulkCheckPresence
0831 13:58:49.749287 (+ 140us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:58:49.749880 (+ 593us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:58:49.750167 (+ 287us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0831 13:58:49.750222 (+ 55us) write_op.cc:312] APPLY: finished
0831 13:58:49.815274 (+ 65052us) log.cc:844] Serialized 34 byte log entry
0831 13:58:49.842451 (+ 27177us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:49.854283 (+ 11832us) write_op.cc:454] Released schema lock
0831 13:58:49.875329 (+ 21046us) write_op.cc:341] FINISH: Updating metrics
0831 13:58:49.930323 (+ 54994us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:58:49.938786 (+ 8463us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4002266,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":270386,"apply.run_cpu_time_us":3004,"apply.run_wall_time_us":263595,"num_ops":3,"prepare.queue_time_us":328026,"prepare.run_cpu_time_us":11754,"prepare.run_wall_time_us":394560,"raft.queue_time_us":184401,"raft.run_cpu_time_us":12828,"raft.run_wall_time_us":333778,"replication_time_us":469260,"thread_start_us":541811,"threads_started":4,"wal-append.queue_time_us":57506,"wal-append.run_cpu_time_us":37167,"wal-append.run_wall_time_us":1941512}]]}
I20250831 13:58:51.426175 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250831 13:58:51.393738 20651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1204) took 1694 ms (client timeout 1752 ms). Trace:
W20250831 13:58:51.426833 20651 rpcz_store.cc:269] 0831 13:58:49.699238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.700105 (+ 867us) service_pool.cc:225] Handling call
0831 13:58:51.391278 (+1691173us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b5400373554. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:51.393686 (+ 2408us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1104503040}
I20250831 13:58:51.329267 22660 consensus_queue.cc:1035] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20250831 13:58:51.428172 22372 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:51.428671 22372 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.429932 22815 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.395839 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.329188 22875 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.430967 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1377) took 1725 ms. Trace:
W20250831 13:58:51.403378 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:51.381088 22856 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.418362 22551 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:51.432703 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.400235 20571 raft_consensus.cc:1273] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:51.437384 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.378737 22801 raft_consensus.cc:695] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:51.378732 22906 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20250831 13:58:51.452075 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.454552 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.405596 22359 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:51.418253 22886 log.cc:927] Time spent T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.196s user 0.001s sys 0.000s
I20250831 13:58:51.481956 20577 raft_consensus.cc:2408] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:58:51.395946 22900 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.506637 22644 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:51.432072 22347 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:51.430487 22372 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:51.511593 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.512882 22819 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:51.534241 20571 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:51.513625 22839 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.517796 20574 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
W20250831 13:58:51.518648 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 68604.7us
I20250831 13:58:51.518941 20569 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250831 13:58:51.524183 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 99234.4us
I20250831 13:58:51.442591 20494 rpcz_store.cc:276] 0831 13:58:49.705680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.812860 (+107180us) service_pool.cc:225] Handling call
0831 13:58:49.813213 (+ 353us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:49.813654 (+ 441us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:49.813703 (+ 49us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:49.814334 (+ 631us) log.cc:844] Serialized 52 byte log entry
0831 13:58:50.267420 (+453086us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:50.292178 (+ 24758us) log.cc:844] Serialized 14 byte log entry
0831 13:58:50.488713 (+196535us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:50.488763 (+ 50us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:50.933237 (+444474us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b4800080058. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c1838c41 00007fb9c182fde7 00007fb9c182fc7a 00007fb9c192506f 00007fb9c1924d37 00007fb9c193935c 00007fb9c1939308 00007fb9c1939288 00007fb9c1939243 00007fb9c19384ec 00007fb9c08b6656
0831 13:58:51.375001 (+441764us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:51.430757 (+ 55756us) raft_consensus.cc:1683] finished
0831 13:58:51.430782 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:51.430915 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1707,"prepare.queue_time_us":24978,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":76649,"raft.run_cpu_time_us":8070,"raft.run_wall_time_us":582321,"spinlock_wait_cycles":17668352,"thread_start_us":631754,"threads_started":2,"wal-append.queue_time_us":524589}
I20250831 13:58:51.431823 22881 raft_consensus.cc:2762] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250831 13:58:51.433163 22837 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:51.516088 22805 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250831 13:58:51.514140 22543 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.413275 22429 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.114s
I20250831 13:58:51.445555 22758 consensus_queue.cc:1046] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
I20250831 13:58:51.562017 22605 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:51.538901 22761 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:51.431470 22836 consensus_queue.cc:237] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.538754 22819 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.547065 22347 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20250831 13:58:51.609710 22839 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.612185 22819 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.616161 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:51.616739 20571 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:51.617025 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.617678 20573 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:58:51.621037 22842 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Append to log took a long time: real 1.206s user 0.004s sys 0.001s
I20250831 13:58:51.624468 20571 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:51.625399 22787 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:51.627216 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 474079us
I20250831 13:58:51.632162 22787 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:51.629254 22687 raft_consensus.cc:3053] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:51.636242 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:51.604780 20616 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:51.637391 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:51.638334 20649 raft_consensus.cc:1273] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:58:51.639178 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.645416 20647 raft_consensus.cc:2408] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:58:51.647524 22787 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:51.648823 20642 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20250831 13:58:51.649535 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 216532us
I20250831 13:58:51.652575 20466 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:51.655520 20467 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:51.656811 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.662915 20643 raft_consensus.cc:1273] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20250831 13:58:51.663393 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 106143us
I20250831 13:58:51.665306 20541 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:51.669759 20466 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:51.672245 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.586501 22881 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20250831 13:58:51.672489 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 906) took 1128 ms. Trace:
I20250831 13:58:51.677877 20466 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:51.677572 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 72883.5us
I20250831 13:58:51.642130 22687 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:51.678087 20496 rpcz_store.cc:276] 0831 13:58:50.544141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:50.977134 (+432993us) service_pool.cc:225] Handling call
0831 13:58:50.977517 (+ 383us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:50.977776 (+ 259us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:50.977813 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:50.978318 (+ 505us) log.cc:844] Serialized 26 byte log entry
0831 13:58:51.043206 (+ 64888us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:51.076492 (+ 33286us) log.cc:844] Serialized 14 byte log entry
0831 13:58:51.076803 (+ 311us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:51.076870 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:51.672262 (+595392us) raft_consensus.cc:1683] finished
0831 13:58:51.672293 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:51.672435 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":37520,"raft.queue_time_us":73,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":18251,"spinlock_wait_cycles":32639872,"thread_start_us":46067,"threads_started":1,"wal-append.queue_time_us":127984}
I20250831 13:58:51.578528 22761 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:51.680127 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.700073 22856 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.700642 20571 raft_consensus.cc:2408] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20250831 13:58:51.700747 22856 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:51.701378 22856 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.703898 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.705945 22381 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:51.707186 22528 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:51.674261 22589 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:51.677278 22787 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:51.712632 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.681496 22794 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:51.713083 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 166061us
I20250831 13:58:51.717347 20616 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:51.711755 22589 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:51.709911 22878 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.681s user 0.003s sys 0.000s
I20250831 13:58:51.680088 22758 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:51.633884 22801 consensus_queue.cc:237] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:51.723480 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 55145.4us
I20250831 13:58:51.714010 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:51.716028 22787 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:51.683845 22808 consensus_queue.cc:1035] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20250831 13:58:51.723111 22589 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:51.683890 22882 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:51.699571 22815 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:51.714797 22860 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:51.720755 22837 consensus_queue.cc:1035] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250831 13:58:51.743049 22787 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:51.735843 22917 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:51.789443 22917 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:51.789800 22917 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:51.790550 22229 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.791728 20651 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 9. Current term is 10. Ops: []
I20250831 13:58:51.792590 22229 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:51.794787 22761 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.777096 22882 raft_consensus.cc:695] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:51.796049 22882 consensus_queue.cc:237] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.800335 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.741751 20468 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
W20250831 13:58:51.761545 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 78571.9us
I20250831 13:58:51.721207 22875 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.709882 22912 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.025s
I20250831 13:58:51.733603 22758 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:51.733070 22681 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:51.741019 22907 log.cc:927] Time spent T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.144s user 0.000s sys 0.000s
I20250831 13:58:51.765682 22822 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20250831 13:58:51.715516 22687 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:51.775380 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 108545us
W20250831 13:58:51.801146 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 59481.2us
I20250831 13:58:51.682559 22372 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:51.807017 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.835204 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:51.835510 20574 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:51.807689 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.836666 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:51.838374 20574 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20250831 13:58:51.820670 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.821588 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:51.760182 22900 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:51.846251 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:51.846472 20502 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:58:51.849188 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:51.850976 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.804836 22758 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:51.856441 20616 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:51.852108 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.855926 20494 raft_consensus.cc:1273] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:51.805191 22794 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:51.836076 20496 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:51.802920 22881 consensus_queue.cc:1046] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20250831 13:58:51.859306 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:51.859870 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:51.802162 22860 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.870136 22922 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:58:51.874354 22614 consensus_meta.cc:287] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.560s user 0.008s sys 0.000s
I20250831 13:58:51.812916 22681 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.885470 20498 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:51.861454 22347 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:51.887688 20571 raft_consensus.cc:2466] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:51.811290 22875 consensus_queue.cc:1035] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20250831 13:58:51.868410 22372 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:58:51.891458 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:51.866211 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:51.896418 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:51.898592 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:51.899257 20501 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:51.908661 21175 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:51.909469 22912 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:51.911170 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:51.911306 22926 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:51.912021 22926 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:51.914323 22908 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.208s user 0.001s sys 0.001s
I20250831 13:58:51.915855 22592 consensus_queue.cc:1046] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:58:51.894145 22347 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.916518 20616 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:51.917546 22592 raft_consensus.cc:3053] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:51.917897 22592 raft_consensus.cc:738] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:51.918953 22660 raft_consensus.cc:2802] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:51.891084 22856 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:51.893100 22372 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:51.922163 20617 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:51.923918 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:51.929466 22917 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.932305 22917 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:51.924855 20501 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:51.918195 22589 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:51.986101 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 63528.9us
I20250831 13:58:51.887245 22614 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:51.921082 22660 raft_consensus.cc:695] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:52.020617 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:51.934551 22372 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:52.043638 20501 raft_consensus.cc:1238] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:51.935978 22881 consensus_queue.cc:1230] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20250831 13:58:52.046977 22899 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.747s user 0.005s sys 0.004s
W20250831 13:58:52.053278 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 131172us
I20250831 13:58:51.918618 22592 consensus_queue.cc:260] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.057258 20499 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
W20250831 13:58:52.052544 22772 consensus_meta.cc:287] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.789s user 0.010s sys 0.000s
W20250831 13:58:52.119966 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:52.121749 22917 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:52.119149 22772 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
W20250831 13:58:52.130227 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 72660.6us
I20250831 13:58:52.130359 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:52.139472 20502 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:52.130618 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:52.054613 22815 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250831 13:58:52.119843 22899 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s user 0.005s sys 0.004s
W20250831 13:58:52.167438 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 340559us
W20250831 13:58:52.179721 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 416153us
I20250831 13:58:52.185089 20466 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:52.195353 22930 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250831 13:58:52.196836 20466 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:52.199929 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:52.200294 22923 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:52.200887 22860 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.204432 22758 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.206431 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:52.207000 20575 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:52.141162 22592 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:52.121357 22787 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.151490 22917 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:52.209535 22925 log.cc:927] Time spent T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.415s user 0.000s sys 0.007s
I20250831 13:58:52.215705 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:52.210060 22681 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.238894 20642 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:52.184563 22347 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.206358 22758 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.236356 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:52.243306 22589 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.191213 22660 consensus_queue.cc:237] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.252171 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:52.253502 22327 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:52.253965 22327 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.254455 22758 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:52.255931 22327 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.258354 20651 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:52.259954 20642 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
I20250831 13:58:52.200228 22872 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:52.273535 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:52.276093 20649 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:52.276914 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:52.279140 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:52.280004 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:52.279874 20541 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:52.263391 22631 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:52.272882 22586 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:52.350332 22946 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250831 13:58:52.350927 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 47993us
I20250831 13:58:52.295850 22758 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:52.357594 20617 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:52.277067 22347 consensus_queue.cc:1046] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20250831 13:58:52.362130 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:52.291739 22881 consensus_queue.cc:1046] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20250831 13:58:52.389819 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:52.439908 22915 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:52.360330 22840 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250831 13:58:52.448911 22915 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:52.356968 22872 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:52.489110 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 132682us
I20250831 13:58:52.356566 22758 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:52.360163 22614 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.358942 22922 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:52.494462 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:52.332582 22631 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:52.348402 22586 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.501361 22758 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:52.504217 22938 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:52.607774 22882 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:52.608000 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 79213.6us
I20250831 13:58:52.608153 22938 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.608376 22882 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.609452 20617 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:52.507210 22787 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.610654 22882 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.507413 22922 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.504622 20497 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:52.513911 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:52.560580 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:52.610538 22938 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.509693 22631 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.359434 22229 consensus_queue.cc:1035] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.139s
I20250831 13:58:52.618121 22229 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:52.611841 20642 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:52.618615 22229 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:52.620052 22229 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:52.632424 22758 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.637449 22937 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:52.640072 22922 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.641202 20497 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:58:52.662220 22917 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:52.665047 22949 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:52.838445 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46464 (request call id 72) took 1266 ms. Trace:
I20250831 13:58:52.847386 22938 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:52.849398 22681 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.051687 22681 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.052529 22758 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.053175 22758 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:52.849790 22542 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.053761 22681 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.053684 22949 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.054010 22542 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.053720 22758 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:52.514024 22372 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.843158 22614 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:52.501783 22872 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:52.847642 22787 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:52.851267 22915 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:52.930784 22589 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:52.954041 20466 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.002s sys 0.018s
W20250831 13:58:52.957283 20496 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.039s sys 0.085s
W20250831 13:58:53.066366 20466 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.720s user 0.002s sys 0.018s
I20250831 13:58:52.939011 22931 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:52.997305 22901 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.008s sys 0.009s
I20250831 13:58:53.067255 22542 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:53.067724 22901 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.734s user 0.008s sys 0.010s
I20250831 13:58:53.066087 22589 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:53.068562 22949 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:53.066474 20496 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s user 0.040s sys 0.085s
I20250831 13:58:53.069914 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1430) took 1015 ms. Trace:
I20250831 13:58:53.070143 22872 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.999881 22965 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:53.070276 22589 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:53.070713 22614 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:53.066381 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 450882us
I20250831 13:58:53.070946 22965 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.065659 22229 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.072782 22614 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.073315 22965 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:52.948971 22586 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.073980 22229 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:53.067301 22372 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:53.070262 20496 rpcz_store.cc:276] 0831 13:58:52.054232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:52.054431 (+ 199us) service_pool.cc:225] Handling call
0831 13:58:52.054738 (+ 307us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:52.054937 (+ 199us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:52.120126 (+ 65189us) log.cc:844] Serialized 14 byte log entry
0831 13:58:53.069583 (+949457us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:53.069636 (+ 53us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:53.069680 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.069756 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:53.069862 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74889,"prepare.queue_time_us":65058,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":79214,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":86750,"thread_start_us":877302,"threads_started":2,"wal-append.queue_time_us":178561}
I20250831 13:58:53.075750 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:53.076460 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:53.026944 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.012s sys 0.033s
W20250831 13:58:53.029006 20502 consensus_meta.cc:287] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.889s user 0.010s sys 0.000s
I20250831 13:58:53.078202 20502 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:58:53.078522 20541 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.010s sys 0.024s
W20250831 13:58:53.033862 20577 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.008s sys 0.012s
W20250831 13:58:53.079075 20541 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.010s sys 0.024s
W20250831 13:58:53.079337 20577 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.730s user 0.008s sys 0.012s
W20250831 13:58:53.061372 20499 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.000s sys 0.012s
W20250831 13:58:53.079917 20499 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.727s user 0.000s sys 0.012s
I20250831 13:58:53.035894 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1409) took 1704 ms. Trace:
I20250831 13:58:53.045197 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.081660 20497 raft_consensus.cc:2366] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:58:53.080586 20570 rpcz_store.cc:276] 0831 13:58:51.331018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.331737 (+ 719us) service_pool.cc:225] Handling call
0831 13:58:51.383481 (+ 51744us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:51.422569 (+ 39088us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.035558 (+1612989us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b54002fe254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:53.035836 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24177920}
I20250831 13:58:53.033367 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1420) took 1207 ms. Trace:
I20250831 13:58:53.048048 20645 rpcz_store.cc:276] 0831 13:58:51.572221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.572500 (+ 279us) service_pool.cc:225] Handling call
0831 13:58:51.572883 (+ 383us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:51.576036 (+ 3153us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:51.576071 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:52.128387 (+552316us) spinlock_profiling.cc:243] Waited 880 us on lock 0x7b2c0003ebe8. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bb3199a1 00007fb9bb328fd2 00007fb9bc9b2021 00007fb9bc9763fa 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32
0831 13:58:52.128794 (+ 407us) log.cc:844] Serialized 26 byte log entry
0831 13:58:52.210926 (+ 82132us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:52.210978 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:52.211066 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:52.624474 (+413408us) raft_consensus.cc:1683] finished
0831 13:58:52.624501 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:52.838372 (+213871us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15294,"raft.queue_time_us":3029,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"spinlock_wait_cycles":2024832,"thread_start_us":81778,"threads_started":1,"wal-append.queue_time_us":161670}
W20250831 13:58:53.052804 22551 consensus_meta.cc:287] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 1.526s user 0.010s sys 0.002s
I20250831 13:58:53.088377 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.056381 22938 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:53.065721 22787 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:53.078773 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.742s user 0.012s sys 0.034s
I20250831 13:58:53.083832 20498 rpcz_store.cc:276] 0831 13:58:51.826486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.827175 (+ 689us) service_pool.cc:225] Handling call
0831 13:58:53.033166 (+1205991us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b54001626d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:53.033300 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":58945408}
I20250831 13:58:53.048843 22959 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:53.060262 21205 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.000s sys 0.012s
I20250831 13:58:53.047438 22937 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:53.093647 22358 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.018s sys 0.025s
I20250831 13:58:53.100039 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:53.076128 22881 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.001s sys 0.014s
I20250831 13:58:53.088047 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.127565 20502 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:53.128595 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.101809 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:53.078392 22592 consensus_meta.cc:287] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.842s user 0.009s sys 0.000s
I20250831 13:58:53.104363 20645 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 9. Current term is 10. Ops: [7.4-9.5]
I20250831 13:58:53.117063 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.110944 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:53.115715 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.104795 20649 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:53.133638 20645 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250831 13:58:53.133592 20498 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:53.128010 22229 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:53.134842 20648 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.008s sys 0.019s
W20250831 13:58:53.135252 20648 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s user 0.008s sys 0.019s
I20250831 13:58:53.135298 22229 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.136590 22968 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:53.126998 22881 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s user 0.001s sys 0.014s
I20250831 13:58:53.142773 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.137897 20496 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:53.101758 22958 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.139528 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1437) took 1010 ms. Trace:
I20250831 13:58:53.148910 20502 rpcz_store.cc:276] 0831 13:58:52.128608 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.138547 (+1009939us) service_pool.cc:225] Handling call
0831 13:58:53.138917 (+ 370us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:53.139137 (+ 220us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:53.139176 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:53.139231 (+ 55us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:53.139274 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.139338 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:53.139461 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:53.140337 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.141079 20644 raft_consensus.cc:1273] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:53.151105 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1441) took 1021 ms. Trace:
I20250831 13:58:53.151573 20502 rpcz_store.cc:276] 0831 13:58:52.129774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.150171 (+1020397us) service_pool.cc:225] Handling call
0831 13:58:53.150489 (+ 318us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:53.150739 (+ 250us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:53.150789 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:53.150846 (+ 57us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:53.150888 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.150956 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:53.151061 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:53.087882 22965 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:58:53.125375 22358 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.020s sys 0.028s
W20250831 13:58:53.103446 21205 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s user 0.000s sys 0.012s
I20250831 13:58:53.144932 22371 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader election won for term 10
I20250831 13:58:53.145032 22881 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:53.145531 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:53.146072 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.109472 22613 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:53.147351 22938 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:53.154209 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:53.099242 22551 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:53.161121 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 58681.5us
I20250831 13:58:53.179931 22919 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:53.160223 21202 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:53.167325 20499 raft_consensus.cc:2366] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:58:53.168370 22937 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:53.169577 22758 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:53.170926 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:53.171772 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:53.154927 22614 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:53.173640 22787 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.153088 22968 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.157519 22881 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.205524 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:53.205744 20568 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:53.187505 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:53.207634 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.161216 20496 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 9. Current term is 10. Ops: []
I20250831 13:58:53.208541 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 919) took 1022 ms. Trace:
I20250831 13:58:53.208864 20496 rpcz_store.cc:276] 0831 13:58:52.185884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.158707 (+972823us) service_pool.cc:225] Handling call
0831 13:58:53.159045 (+ 338us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:53.208286 (+ 49241us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.208483 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:53.209771 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:53.210263 20496 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:53.188952 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:53.211351 20645 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:53.189769 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.185616 20642 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:53.198424 20647 raft_consensus.cc:3053] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:53.106204 22959 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.067548 22931 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.214955 20647 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:53.162478 22965 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.199337 22229 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:53.218917 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.176504 22773 raft_consensus.cc:3053] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 9 LEADER]: Stepping down as leader of term 9
I20250831 13:58:53.229167 22773 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:53.220304 20647 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.229954 22773 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.222570 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.223186 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.180631 22613 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.219342 22100 raft_consensus.cc:3053] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:53.203341 22614 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.205741 21208 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:53.233031 20642 raft_consensus.cc:2391] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:53.233170 20645 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:53.238456 22881 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.242029 20499 raft_consensus.cc:2391] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:53.156786 22958 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.230557 22968 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.246485 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.256599 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:53.261487 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:53.262086 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:53.248083 22773 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:53.266989 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.238468 22551 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.264175 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.270618 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.271977 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.261054 22968 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:53.199011 22919 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.245541 22100 raft_consensus.cc:738] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:58:53.273839 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 113210us
I20250831 13:58:53.274801 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.275527 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.277041 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.278146 20466 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:53.285140 20541 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:53.285866 20651 raft_consensus.cc:2408] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 13:58:53.272637 20499 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:53.279978 20496 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:58:53.312613 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.313643 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.282176 20617 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:53.317587 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:53.279138 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:53.280082 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.317463 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:53.286170 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.320324 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.321281 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:53.317458 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:53.322213 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:53.323412 20573 raft_consensus.cc:2408] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:53.311740 22808 raft_consensus.cc:2802] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:53.282007 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47746 (request call id 79) took 2148 ms. Trace:
I20250831 13:58:53.328197 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:53.328831 20571 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:53.329332 20496 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20250831 13:58:53.329882 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 923) took 1136 ms. Trace:
I20250831 13:58:53.326130 20642 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:53.330492 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:53.283834 22551 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:53.317747 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:53.331084 20573 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:53.330237 20496 rpcz_store.cc:276] 0831 13:58:52.193911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.328860 (+1134949us) service_pool.cc:225] Handling call
0831 13:58:53.329147 (+ 287us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:53.329678 (+ 531us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.329834 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:53.332448 20466 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:53.312561 22937 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:53.344025 20466 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:53.345922 22851 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:58:53.346127 22967 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20250831 13:58:53.282723 22968 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.326404 22969 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:53.329983 22917 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:53.287326 22758 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20250831 13:58:53.352805 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:53.330199 22881 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.363667 20617 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:53.334622 22808 raft_consensus.cc:695] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:53.332976 22551 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:53.342429 22371 raft_consensus.cc:695] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 10 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:53.352850 22851 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:53.380674 20573 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:58:53.312045 22973 consensus_queue.cc:1046] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.081s
I20250831 13:58:53.428272 22808 consensus_queue.cc:237] T e9d8af1af01e4b7f83bc2aaded81ccbf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.381611 22551 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:53.370085 22917 raft_consensus.cc:695] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:53.428393 22979 consensus_queue.cc:1046] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20250831 13:58:53.435518 22613 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.440652 22906 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:58:53.346837 22100 consensus_queue.cc:260] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.448844 22954 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.449077 22971 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:53.449491 22954 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.449738 22971 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:53.485498 22906 raft_consensus.cc:695] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:53.427021 22371 consensus_queue.cc:237] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.431341 22542 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.377775 22881 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.363297 22969 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.442771 22965 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.551244 22906 consensus_queue.cc:237] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.487569 22758 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.536321 22614 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.554958 22965 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.557503 22931 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:53.585464 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:53.558821 22959 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.547215 22542 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.560237 22958 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.570581 22919 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.571815 22969 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.668072 22971 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.682889 22974 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:53.663282 22931 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:53.695571 22881 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.695278 22931 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.683524 22974 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:53.699143 22883 log.cc:927] Time spent T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 2.664s user 0.022s sys 0.024s
I20250831 13:58:53.680342 22955 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:53.666482 22959 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:53.594527 22965 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.704159 22874 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.328605 21205 rpcz_store.cc:276] 0831 13:58:51.131733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.132696 (+ 963us) service_pool.cc:225] Handling call
0831 13:58:51.799166 (+666470us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b58000c0098. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:51.799395 (+ 229us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:51.799659 (+ 264us) raft_consensus.cc:1532] Early marking committed up to index 8
0831 13:58:52.122254 (+322595us) log.cc:844] Serialized 14 byte log entry
0831 13:58:53.247104 (+1124850us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:53.248750 (+ 1646us) raft_consensus.cc:1643] Marking committed up to 8
0831 13:58:53.253015 (+ 4265us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.261162 (+ 8147us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:53.270021 (+ 8859us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":382943,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":175590,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11142,"spinlock_wait_cycles":23210880,"thread_start_us":1246068,"threads_started":3,"wal-append.queue_time_us":949818}
I20250831 13:58:53.663049 22861 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.570788 22758 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:53.697641 22931 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.670622 22968 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.718277 22959 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.704706 22874 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:53.695575 22542 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.730211 22974 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.732088 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1252) took 1536 ms. Trace:
I20250831 13:58:53.732609 20644 rpcz_store.cc:276] 0831 13:58:52.195745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:52.198336 (+ 2591us) service_pool.cc:225] Handling call
0831 13:58:52.198656 (+ 320us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:53.151174 (+952518us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.731848 (+580674us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b540028bc54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:53.732021 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2160543104}
I20250831 13:58:53.719028 22955 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:53.734202 22955 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:53.705247 22965 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.584990 22917 consensus_queue.cc:237] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.747503 22998 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.748198 22998 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:53.752097 22998 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.765256 22554 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.765823 22554 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.767992 22554 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.768806 22695 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.769451 22695 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.771689 22695 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.720269 22861 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:53.727523 22954 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:53.815807 22910 log.cc:927] Time spent T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 2.209s user 0.010s sys 0.024s
W20250831 13:58:53.823896 22992 log.cc:927] Time spent T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20250831 13:58:53.825209 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 925) took 1630 ms. Trace:
I20250831 13:58:53.825552 22758 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:53.825594 20499 rpcz_store.cc:276] 0831 13:58:52.194496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.350990 (+1156494us) service_pool.cc:225] Handling call
0831 13:58:53.351374 (+ 384us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:53.370375 (+ 19001us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:53.370410 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:53.372424 (+ 2014us) log.cc:844] Serialized 26 byte log entry
0831 13:58:53.761882 (+389458us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:53.762207 (+ 325us) log.cc:844] Serialized 14 byte log entry
0831 13:58:53.762406 (+ 199us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:53.762475 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:53.824957 (+ 62482us) raft_consensus.cc:1683] finished
0831 13:58:53.824980 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:53.825151 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19096,"prepare.queue_time_us":560,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":18689,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":9709,"thread_start_us":324376,"threads_started":1,"wal-append.queue_time_us":339764}
I20250831 13:58:53.828998 22959 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.829638 20499 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20250831 13:58:53.835939 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 491598us
I20250831 13:58:53.842931 22861 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:53.849772 20497 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.001s sys 0.009s
W20250831 13:58:53.850113 20497 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.001s sys 0.009s
I20250831 13:58:53.832901 22874 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:54.247057 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1421) took 2419 ms. Trace:
I20250831 13:58:54.281699 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 924) took 2087 ms. Trace:
I20250831 13:58:54.284452 20646 raft_consensus.cc:3058] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:54.288622 20496 rpcz_store.cc:276] 0831 13:58:52.194213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.335540 (+1141327us) service_pool.cc:225] Handling call
0831 13:58:53.335939 (+ 399us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:53.347771 (+ 11832us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:53.347815 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:53.351541 (+ 3726us) log.cc:844] Serialized 26 byte log entry
0831 13:58:53.845452 (+493911us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:54.281046 (+435594us) log.cc:844] Serialized 14 byte log entry
0831 13:58:54.281271 (+ 225us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.281358 (+ 87us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:54.281413 (+ 55us) raft_consensus.cc:1683] finished
0831 13:58:54.281429 (+ 16us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:54.281623 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6085,"raft.queue_time_us":7146,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":6797,"thread_start_us":422814,"threads_started":1,"wal-append.queue_time_us":698051}
W20250831 13:58:54.173645 20617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.802s user 0.015s sys 0.023s
I20250831 13:58:54.291231 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:54.291338 20617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.920s user 0.015s sys 0.031s
W20250831 13:58:54.292485 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 928855us
I20250831 13:58:54.298772 20617 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:54.177381 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.005s sys 0.020s
W20250831 13:58:54.302211 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.716s user 0.006s sys 0.025s
I20250831 13:58:54.303207 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.303393 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.303277 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.304234 20645 raft_consensus.cc:2366] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:54.015041 21708 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.304411 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.305135 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:53.851082 20498 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.000s sys 0.010s
W20250831 13:58:54.306106 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:54.306327 20498 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.126s user 0.011s sys 0.021s
W20250831 13:58:53.940207 20502 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.002s sys 0.029s
I20250831 13:58:54.303262 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.861533 22937 raft_consensus.cc:695] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:58:54.307129 20502 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.992s user 0.020s sys 0.038s
I20250831 13:58:54.306089 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.311775 20496 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:54.312204 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 917) took 2128 ms. Trace:
I20250831 13:58:54.312631 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.290778 20493 rpcz_store.cc:276] 0831 13:58:51.827622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.827769 (+ 147us) service_pool.cc:225] Handling call
0831 13:58:51.858411 (+ 30642us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:51.900374 (+ 41963us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:51.900407 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:51.901038 (+ 631us) log.cc:844] Serialized 52 byte log entry
0831 13:58:52.161800 (+260762us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:52.193249 (+ 31449us) log.cc:844] Serialized 14 byte log entry
0831 13:58:52.272988 (+ 79739us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.246495 (+1973507us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b54001284d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:54.246566 (+ 71us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:54.246634 (+ 68us) raft_consensus.cc:1683] finished
0831 13:58:54.246762 (+ 128us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:54.246967 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":175468,"prepare.queue_time_us":169817,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":85917,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":15367,"spinlock_wait_cycles":85478656,"thread_start_us":123012,"threads_started":1,"wal-append.queue_time_us":320501,"wal-append.run_cpu_time_us":2221,"wal-append.run_wall_time_us":1003778}
W20250831 13:58:54.306924 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:54.304384 20644 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:54.304932 21708 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:53.846145 21174 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.502s user 0.005s sys 0.025s
I20250831 13:58:54.316078 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.318293 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:54.176813 22808 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s user 0.010s sys 0.049s
W20250831 13:58:54.289124 20543 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.527s (SENT) [suppressed 18 similar messages]
I20250831 13:58:54.319567 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.316962 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.320663 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1411) took 2940 ms. Trace:
I20250831 13:58:54.035637 22879 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:54.303809 21176 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.002s sys 0.018s
I20250831 13:58:53.892406 22782 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:54.245590 22955 consensus_meta.cc:287] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.002s
W20250831 13:58:54.303228 22906 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.003s sys 0.011s
I20250831 13:58:54.013331 22881 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.323417 21708 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.995508 22773 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.326030 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:54.321964 22973 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.004s sys 0.032s
I20250831 13:58:54.326179 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.326409 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.958431 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:54.323288 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.231817 22926 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:54.206383 22987 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.958050 22778 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:54.324443 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:53.958842 22959 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:53.990016 22577 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:53.892196 22615 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.328326 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.418895 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.419219 20570 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:58:54.054318 22758 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.328403 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.144618 22993 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.011986 22974 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.077579 22986 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.041940 22968 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:54.330677 21176 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s user 0.002s sys 0.018s
I20250831 13:58:54.428632 22968 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:54.323882 22808 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.880s user 0.010s sys 0.055s
I20250831 13:58:54.321521 22966 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.431372 22968 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:53.919226 22551 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:54.374187 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1246) took 2709 ms. Trace:
I20250831 13:58:54.441037 20650 rpcz_store.cc:276] 0831 13:58:51.664665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.667099 (+ 2434us) service_pool.cc:225] Handling call
0831 13:58:51.667347 (+ 248us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:51.667550 (+ 203us) raft_consensus.cc:1532] Early marking committed up to index 6
0831 13:58:51.673173 (+ 5623us) log.cc:844] Serialized 14 byte log entry
0831 13:58:53.831716 (+2158543us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b18004712e0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba2da 00007fb9bc9809da
0831 13:58:54.373698 (+541982us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:54.373766 (+ 68us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:54.373852 (+ 86us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.373957 (+ 105us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:54.374111 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70147,"prepare.queue_time_us":3214,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":78775,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":117,"spinlock_wait_cycles":4420352,"thread_start_us":471512,"threads_started":1,"wal-append.queue_time_us":2716575}
I20250831 13:58:54.374909 20541 leader_election.cc:400] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:58:54.446852 20541 leader_election.cc:403] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
W20250831 13:58:54.321630 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 992781us
W20250831 13:58:53.935866 22921 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 2.132s user 0.010s sys 0.026s
I20250831 13:58:54.380582 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.382198 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:54.382740 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 89525.7us
I20250831 13:58:54.382906 20494 raft_consensus.cc:2391] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:54.374046 22830 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:54.383529 22359 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 13:58:54.370523 22906 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s user 0.003s sys 0.011s
I20250831 13:58:54.383759 22937 consensus_queue.cc:237] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.384991 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1253) took 2188 ms. Trace:
I20250831 13:58:54.386572 20647 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:54.372103 22955 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:54.386955 22973 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.946s user 0.004s sys 0.032s
I20250831 13:58:54.392212 20646 raft_consensus.cc:1273] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250831 13:58:54.517032 20646 pending_rounds.cc:85] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Aborting all ops after (but not including) 3
I20250831 13:58:54.517287 20646 pending_rounds.cc:107] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250831 13:58:54.517493 20646 raft_consensus.cc:2887] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 13:58:54.387580 22877 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 2.689s user 0.020s sys 0.000s
W20250831 13:58:54.376164 22978 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20250831 13:58:54.408591 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:54.375273 22528 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:54.395243 22592 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:54.332867 22879 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.371140 22782 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:54.426210 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 137110us
I20250831 13:58:54.395076 22899 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:54.387821 22605 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20250831 13:58:54.407503 21208 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:54.505788 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.506299 20493 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:54.506258 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.528105 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.424198 23000 consensus_queue.cc:1035] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20250831 13:58:54.421880 22577 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.387883 22773 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.508172 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.535100 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.425626 22758 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.396896 22881 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.538247 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.539254 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.539788 22881 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:54.512979 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 151129us
I20250831 13:58:54.540864 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:54.415200 22778 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.518810 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1218) took 4093 ms. Trace:
I20250831 13:58:54.542872 20646 rpcz_store.cc:276] 0831 13:58:50.425210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:50.425446 (+ 236us) service_pool.cc:225] Handling call
0831 13:58:50.425701 (+ 255us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:51.349738 (+924037us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b58008c39b0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc8f1003 00007fb9bc954577 00007fb9bc96cfa7 00007fb9bc96e79e 00007fb9bc978af9 00007fb9bc9793c6 00007fb9bc975d4f 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44
0831 13:58:54.228312 (+2878574us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b58008c3cb0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc8f1003 00007fb9bc954577 00007fb9bc96cfa7 00007fb9bc96e79e 00007fb9bc978af9 00007fb9bc9793c6 00007fb9bc975d4f 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44
0831 13:58:54.518551 (+290239us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.518755 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3337805312}
I20250831 13:58:54.543998 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.509356 22808 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:54.512639 20643 rpcz_store.cc:276] 0831 13:58:52.196096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:52.196893 (+ 797us) service_pool.cc:225] Handling call
0831 13:58:52.197229 (+ 336us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:52.197557 (+ 328us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:52.197589 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:53.704843 (+1507254us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b2c0003ebe8. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bb3199a1 00007fb9bb328fd2 00007fb9bc9b2021 00007fb9bc9763fa 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32
0831 13:58:53.705242 (+ 399us) log.cc:844] Serialized 26 byte log entry
0831 13:58:54.167186 (+461944us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:54.167244 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.167337 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:54.384650 (+217313us) raft_consensus.cc:1683] finished
0831 13:58:54.384674 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:54.384922 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1376,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"spinlock_wait_cycles":29834496,"thread_start_us":451179,"threads_started":1,"wal-append.queue_time_us":574584}
I20250831 13:58:54.552981 22551 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:54.522923 22528 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.527217 22592 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.505707 22986 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:54.521528 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.510792 22359 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:54.416910 22987 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.507228 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:54.518734 23008 log.cc:927] Time spent T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20250831 13:58:54.516711 22968 raft_consensus.cc:3058] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:58:54.531541 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.431886 22966 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.535032 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.422895 22959 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.425552 22615 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:54.510424 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 106370us
W20250831 13:58:54.523481 22342 raft_consensus.cc:934] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20250831 13:58:54.467630 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.467245 22974 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.854214 20570 raft_consensus.cc:2408] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 13:58:54.855410 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.524994 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1439) took 2395 ms. Trace:
I20250831 13:58:54.509155 22993 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.856027 22999 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:54.876473 20577 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:54.761893 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:54.851195 20493 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
W20250831 13:58:54.878366 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:54.723019 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1445) took 1531 ms. Trace:
I20250831 13:58:54.879065 20499 rpcz_store.cc:276] 0831 13:58:53.191838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.844202 (+652364us) service_pool.cc:225] Handling call
0831 13:58:53.910309 (+ 66107us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:53.931166 (+ 20857us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:53.931280 (+ 114us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:54.316085 (+384805us) log.cc:844] Serialized 26 byte log entry
0831 13:58:54.427875 (+111790us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:54.427911 (+ 36us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.428015 (+ 104us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:54.722741 (+294726us) raft_consensus.cc:1683] finished
0831 13:58:54.722770 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:54.722940 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":28092,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"thread_start_us":56718,"threads_started":1,"wal-append.queue_time_us":193993}
I20250831 13:58:54.881060 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.723536 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.854143 22897 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:54.539909 20650 raft_consensus.cc:2391] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:54.555528 20571 raft_consensus.cc:1273] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:54.872237 20496 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:54.565449 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.828763 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:54.874938 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 325784us
I20250831 13:58:54.700506 20647 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250831 13:58:54.856590 20498 rpcz_store.cc:276] 0831 13:58:52.129227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.140874 (+1011647us) service_pool.cc:225] Handling call
0831 13:58:53.147148 (+ 6274us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:53.152981 (+ 5833us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:53.153009 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:53.154063 (+ 1054us) log.cc:844] Serialized 52 byte log entry
0831 13:58:54.307111 (+1153048us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:54.524283 (+217172us) log.cc:844] Serialized 14 byte log entry
0831 13:58:54.524571 (+ 288us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.524668 (+ 97us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:54.524721 (+ 53us) raft_consensus.cc:1683] finished
0831 13:58:54.524739 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:54.524919 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18701,"prepare.queue_time_us":77698,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":5570,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":24022,"thread_start_us":1318117,"threads_started":2,"wal-append.queue_time_us":1136394}
I20250831 13:58:54.925382 20543 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:54.939174 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.940045 20499 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:54.941154 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.947662 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.668740 22881 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:54.949330 20570 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:58:54.563585 23007 log.cc:927] Time spent T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.139s user 0.000s sys 0.000s
I20250831 13:58:54.554426 22782 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:54.950313 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 433860us
I20250831 13:58:54.924074 20647 pending_rounds.cc:85] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Aborting all ops after (but not including) 3
W20250831 13:58:54.951859 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:54.950266 20645 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:54.987176 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.994553 20647 pending_rounds.cc:107] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250831 13:58:54.980474 20643 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:58:54.988871 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 147789us
I20250831 13:58:54.716850 22955 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:54.995115 20647 raft_consensus.cc:2887] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:58:54.995237 20576 raft_consensus.cc:2408] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:54.714408 22605 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:54.998790 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.990350 20573 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:58:54.999593 20576 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:55.000691 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:54.668107 22790 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:54.992986 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:55.006392 20498 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:54.992935 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.007234 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:55.007830 20642 raft_consensus.cc:2408] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:54.553138 22778 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.010481 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:54.942440 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.876956 22359 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:54.896040 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:54.995886 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1264) took 1798 ms. Trace:
I20250831 13:58:55.072115 20647 rpcz_store.cc:276] 0831 13:58:53.197904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.198025 (+ 121us) service_pool.cc:225] Handling call
0831 13:58:53.198301 (+ 276us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:54.386334 (+1188033us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b5800425cb0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc8f1003 00007fb9bc954577 00007fb9bc96cfa7 00007fb9bc96e79e 00007fb9bc978af9 00007fb9bc9793c6 00007fb9bc975d4f 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44
0831 13:58:54.995654 (+609320us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.995842 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1387619072}
I20250831 13:58:54.996217 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.073745 20499 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 9.
I20250831 13:58:54.996866 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.879200 22229 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:54.991564 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:54.987761 22551 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:54.875922 22999 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:55.078189 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.079422 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.080372 20647 raft_consensus.cc:2408] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20250831 13:58:55.015811 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.009721 22605 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:55.017645 23013 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:54.989547 22881 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:55.018651 22592 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:55.014835 22790 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:55.019593 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1421) took 3626 ms. Trace:
I20250831 13:58:55.019608 22528 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.035403 22577 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:54.323602 20572 rpcz_store.cc:276] 0831 13:58:51.379978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.380293 (+ 315us) service_pool.cc:225] Handling call
0831 13:58:51.396338 (+ 16045us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:51.396530 (+ 192us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:51.409473 (+ 12943us) log.cc:844] Serialized 14 byte log entry
0831 13:58:54.193826 (+2784353us) spinlock_profiling.cc:243] Waited 1.34 ms on lock 0x7b18002a8ce0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba2da 00007fb9bc9809da
0831 13:58:54.320330 (+126504us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:54.320384 (+ 54us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:54.320439 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.320504 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:54.320610 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5995,"prepare.queue_time_us":12897,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":1960,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":3992,"spinlock_wait_cycles":3089792,"thread_start_us":124374,"threads_started":1}
W20250831 13:58:54.992888 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 43491.9us
I20250831 13:58:54.564235 22808 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:55.045616 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.052336 20577 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:55.053258 22615 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.065882 22987 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.067200 22959 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.068354 22973 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:55.068764 21174 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:54.988186 22897 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:54.536676 22899 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:55.340404 20647 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:55.336661 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.412962 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1015) took 1064 ms. Trace:
W20250831 13:58:55.413396 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:55.413372 20647 rpcz_store.cc:276] 0831 13:58:54.348659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.339845 (+991186us) service_pool.cc:225] Handling call
0831 13:58:55.340183 (+ 338us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:55.412584 (+ 72401us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:55.412788 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:55.414453 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:55.347532 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 155491us
I20250831 13:58:55.415540 20647 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
I20250831 13:58:55.416085 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1016) took 1038 ms. Trace:
W20250831 13:58:55.379119 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:55.111768 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.416669 20568 rpcz_store.cc:276] 0831 13:58:51.393616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.394705 (+ 1089us) service_pool.cc:225] Handling call
0831 13:58:51.394969 (+ 264us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:53.206278 (+1811309us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:55.019364 (+1813086us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b54001a6dd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:55.019535 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4164055424}
I20250831 13:58:55.417920 20647 rpcz_store.cc:276] 0831 13:58:54.377466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.415126 (+1037660us) service_pool.cc:225] Handling call
0831 13:58:55.415382 (+ 256us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:55.415853 (+ 471us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:55.416023 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:55.418573 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 493301us
W20250831 13:58:55.417709 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 417000us
I20250831 13:58:55.420573 20647 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
W20250831 13:58:55.414500 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:55.420763 20543 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:55.422317 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1017) took 1044 ms. Trace:
I20250831 13:58:55.422960 20543 leader_election.cc:400] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
W20250831 13:58:55.416769 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 409610us
I20250831 13:58:55.422960 20647 rpcz_store.cc:276] 0831 13:58:54.377806 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.420176 (+1042370us) service_pool.cc:225] Handling call
0831 13:58:55.420415 (+ 239us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:55.421853 (+ 1438us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:55.422006 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:55.424355 20543 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:55.431530 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.432478 20568 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:55.433233 20616 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:55.433876 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.434743 20568 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:55.436151 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.436908 20568 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:55.438534 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.439383 20568 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:55.498764 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:55.499840 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 72754.4us
W20250831 13:58:55.183012 22932 log.cc:927] Time spent T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 2.902s user 0.014s sys 0.026s
I20250831 13:58:55.447921 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.501037 20493 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20250831 13:58:55.499568 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 66399.5us
I20250831 13:58:55.506031 20643 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:55.529048 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.529903 20572 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:55.533197 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1293) took 1006 ms. Trace:
I20250831 13:58:55.533811 20568 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:55.533571 20646 rpcz_store.cc:276] 0831 13:58:54.526470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.439264 (+912794us) service_pool.cc:225] Handling call
0831 13:58:55.439617 (+ 353us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:55.532871 (+ 93254us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:55.532902 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:55.532946 (+ 44us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:55.532978 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:55.533054 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:55.533145 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":92920}
W20250831 13:58:55.521376 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 92501.6us
W20250831 13:58:55.537653 20496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.000s sys 0.023s
W20250831 13:58:55.538075 20496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s user 0.000s sys 0.023s
I20250831 13:58:55.541745 20642 raft_consensus.cc:1238] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:55.543896 20493 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20250831 13:58:55.377453 22904 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 3.637s user 0.019s sys 0.076s
I20250831 13:58:55.552850 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.553612 20568 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:55.554738 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.555470 20568 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:58:55.556486 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.557152 20568 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:55.558117 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.497551 22772 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:55.118978 22993 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.376968 22923 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:55.427798 22881 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:55.376753 22849 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:55.589517 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.590178 20572 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:55.504046 22991 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:55.497743 22543 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:55.524652 22899 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:55.098524 22551 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:55.563751 20617 connection.cc:232] RPC call timeout handler was delayed by -2.1941934049999645s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1654s for a response
W20250831 13:58:55.588523 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 89785.2us
I20250831 13:58:55.098671 22229 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:55.500196 22959 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:55.560165 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.437422 22577 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:55.595160 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 94378.7us
I20250831 13:58:55.562199 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:55.188869 22986 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:55.327744 22359 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:55.581912 22999 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.550804 23020 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20250831 13:58:55.554858 22904 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.815s user 0.024s sys 0.102s
W20250831 13:58:55.703501 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 279200us
W20250831 13:58:55.625039 20616 connection.cc:232] RPC call timeout handler was delayed by -2.08448850100001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
W20250831 13:58:55.705338 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:55.541579 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 487408us
I20250831 13:58:55.642158 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:55.643098 20568 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 3.
I20250831 13:58:55.707278 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.708045 20568 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:55.709007 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.612448 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.606244 23013 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:55.533833 22973 consensus_queue.cc:1046] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.080s
I20250831 13:58:55.648694 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1442) took 3799 ms. Trace:
I20250831 13:58:55.856019 20574 rpcz_store.cc:276] 0831 13:58:51.849492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.850119 (+ 627us) service_pool.cc:225] Handling call
0831 13:58:51.850413 (+ 294us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:51.861663 (+ 11250us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:51.861691 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:51.867340 (+ 5649us) log.cc:844] Serialized 26 byte log entry
0831 13:58:52.031539 (+164199us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:52.031606 (+ 67us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:52.031700 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:55.182323 (+3150623us) spinlock_profiling.cc:243] Waited 8.39 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:55.642745 (+460422us) raft_consensus.cc:1683] finished
0831 13:58:55.642779 (+ 34us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:55.648616 (+ 5837us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10900,"raft.queue_time_us":47192,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":4953,"spinlock_wait_cycles":19292416,"thread_start_us":139192,"threads_started":1,"wal-append.queue_time_us":295438}
W20250831 13:58:55.700831 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:55.701381 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 57423.3us
I20250831 13:58:55.704620 20543 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:55.709662 20568 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 1.
I20250831 13:58:55.860668 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.861343 20568 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:55.595711 22528 consensus_queue.cc:1035] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.105s
I20250831 13:58:55.858417 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.867689 23017 consensus_queue.cc:1046] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.165s
I20250831 13:58:55.872077 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1447) took 2679 ms. Trace:
I20250831 13:58:55.872488 20496 rpcz_store.cc:276] 0831 13:58:53.192420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:54.986876 (+1794456us) service_pool.cc:225] Handling call
0831 13:58:55.010257 (+ 23381us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:55.010603 (+ 346us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:55.010645 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:55.016844 (+ 6199us) log.cc:844] Serialized 52 byte log entry
0831 13:58:55.538523 (+521679us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:55.607884 (+ 69361us) log.cc:844] Serialized 14 byte log entry
0831 13:58:55.610751 (+ 2867us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:55.610826 (+ 75us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:55.871865 (+261039us) raft_consensus.cc:1683] finished
0831 13:58:55.871883 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:55.872021 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":163079,"raft.queue_time_us":18850,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":8676,"thread_start_us":547696,"threads_started":2,"wal-append.queue_time_us":840058}
I20250831 13:58:55.875348 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.872686 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.886262 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.887173 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1355) took 1061 ms. Trace:
I20250831 13:58:55.887547 20568 rpcz_store.cc:276] 0831 13:58:54.826146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.885986 (+1059840us) service_pool.cc:225] Handling call
0831 13:58:55.887109 (+ 1123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:55.888700 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:55.889346 20568 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:58:55.891584 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 187052us
I20250831 13:58:55.907217 20543 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:55.601791 22849 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:55.645918 22987 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.239s
I20250831 13:58:55.712286 22940 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:55.647749 22577 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:55.640064 22993 consensus_queue.cc:1046] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.258s
I20250831 13:58:55.925845 22904 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:55.891443 22986 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:55.927251 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1356) took 1002 ms. Trace:
I20250831 13:58:55.960286 20568 rpcz_store.cc:276] 0831 13:58:54.924970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.926522 (+1001552us) service_pool.cc:225] Handling call
0831 13:58:55.926803 (+ 281us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:55.926964 (+ 161us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:55.926985 (+ 21us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:55.927024 (+ 39us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:55.927064 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:55.927115 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:55.927202 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:55.961894 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:55.639605 23016 consensus_queue.cc:1046] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.271s
W20250831 13:58:55.932111 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 72444.2us
I20250831 13:58:55.873862 22790 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.609726 22923 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:55.959436 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 50923.8us
I20250831 13:58:55.972819 20467 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:56.012518 20543 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.008131 20568 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:56.019587 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 46883.4us
I20250831 13:58:56.020556 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:55.945801 22999 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:55.959357 22973 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:56.025646 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:56.031348 20543 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:56.033679 20467 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:56.036888 20467 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:56.011459 21202 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:56.010371 22808 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:55.873246 23013 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:56.044366 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 339731us
W20250831 13:58:56.044736 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:56.009791 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 46252.1us
I20250831 13:58:56.032582 22790 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:55.970049 22904 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.053020 22904 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:56.037968 23016 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:56.036586 22772 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:56.027667 23022 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:56.054615 20543 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:55.963546 23027 consensus_queue.cc:1046] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.115s
I20250831 13:58:56.055958 22605 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:56.020251 22993 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20250831 13:58:56.039330 22853 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:56.055981 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.065701 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:55.958194 22940 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.073287 20543 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:56.050257 22808 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:56.086100 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.056810 22229 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:56.044118 22543 consensus_queue.cc:1046] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
W20250831 13:58:56.071036 22968 consensus_meta.cc:287] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 1.171s user 0.009s sys 0.002s
W20250831 13:58:56.090442 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 78104.4us
I20250831 13:58:56.060284 22772 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.092985 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.092257 20643 raft_consensus.cc:2391] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:58:56.093616 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 56825.8us
I20250831 13:58:56.094378 20642 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:56.094879 20574 raft_consensus.cc:2408] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 13:58:56.095209 20467 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.095866 20543 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:56.097074 23025 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.411s user 0.000s sys 0.000s
I20250831 13:58:56.072134 23036 consensus_queue.cc:1046] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20250831 13:58:56.096339 20467 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:56.098726 22999 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:56.099026 23022 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:56.037375 22973 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.100075 20467 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.055963 23015 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:56.102114 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.102567 20467 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.107436 20543 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.087988 22986 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:56.110105 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.110785 20467 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.087127 22359 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:56.090369 22371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s user 0.000s sys 0.007s
I20250831 13:58:56.115353 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.116007 20543 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.114846 20467 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.094586 22772 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:56.093117 22551 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:56.093703 22968 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
W20250831 13:58:56.118992 23029 log.cc:927] Time spent T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.229s user 0.001s sys 0.000s
I20250831 13:58:56.119199 23038 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:56.119782 20543 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:56.113900 22993 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:56.133378 20543 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:58:56.121801 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:56.110574 22808 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:56.122622 20467 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:58:56.123093 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 100693us
W20250831 13:58:56.136482 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:56.111677 23022 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:56.130623 20568 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:56.067430 21176 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:56.117286 22371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s user 0.000s sys 0.007s
I20250831 13:58:56.136658 20467 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.117626 22853 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:56.121014 22849 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:56.120769 22347 raft_consensus.cc:3053] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:58:56.139547 20543 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:56.123927 22986 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:56.126789 23027 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:56.154047 20543 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.135586 22973 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:56.137970 23022 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.151492 23043 raft_consensus.cc:3053] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:56.151659 21202 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:56.152050 23037 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:56.153482 23036 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:56.147925 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.135603 22808 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.156802 20543 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.160600 20576 raft_consensus.cc:2466] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:56.144021 23015 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 13:58:56.144598 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 133560us
I20250831 13:58:56.140512 23038 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.143074 22853 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.148639 22968 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:56.164994 22914 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:56.156919 22999 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:56.145011 22347 raft_consensus.cc:738] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:56.149603 22986 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:56.154235 22605 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:56.167780 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:56.168944 20467 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.185513 20467 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:56.186431 20467 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.157469 22993 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:56.154341 23027 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.188706 20467 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.189817 20467 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.171372 22780 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250831 13:58:56.165879 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 115838us
I20250831 13:58:56.172379 23038 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:56.179306 22853 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:56.173061 22861 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:58:56.176400 20468 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:56.164222 23022 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:56.170226 22808 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:56.191401 21202 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:56.168733 23043 raft_consensus.cc:738] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:56.187904 22605 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.168641 23037 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.202126 22528 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:56.174897 21210 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:56.203091 23046 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:56.173511 22577 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:56.180068 22968 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:56.206362 20543 connection.cc:232] RPC call timeout handler was delayed by -1.3580350359999898s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1847s for a response
I20250831 13:58:56.184770 22986 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:56.180778 22999 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.200380 22872 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:56.200919 22551 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:56.170883 23015 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.202718 22780 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:56.188078 22993 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:56.199139 22347 consensus_queue.cc:260] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:56.268339 21202 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:58:56.268610 22918 log.cc:927] Time spent T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 4.477s user 0.016s sys 0.046s
W20250831 13:58:56.270413 22909 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 4.656s user 0.019s sys 0.050s
W20250831 13:58:56.201849 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 53279.6us
I20250831 13:58:56.254205 22772 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:56.260033 22881 consensus_meta.cc:287] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.649s user 0.010s sys 0.000s
W20250831 13:58:56.242278 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 52559.5us
I20250831 13:58:56.274313 20467 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.246860 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.276252 20467 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.277635 20467 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:56.248883 22794 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:56.255321 21204 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:56.269079 22780 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:56.258803 20468 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:56.272975 22347 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:56.294188 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1394) took 5874 ms. Trace:
I20250831 13:58:56.286482 21204 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:56.281147 22849 raft_consensus.cc:2802] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250831 13:58:56.281441 22794 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.241084 22605 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:56.209935 22987 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:56.242235 22528 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.259536 22872 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.242023 23037 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:56.280800 22881 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.303027 22794 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:56.248219 22968 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:58:56.259860 22991 consensus_meta.cc:287] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.650s user 0.008s sys 0.001s
I20250831 13:58:56.271389 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1334) took 4727 ms. Trace:
I20250831 13:58:56.309654 22528 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:56.313292 22872 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:56.259402 22999 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:56.204691 22923 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:56.205168 22861 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:56.271104 23043 consensus_queue.cc:260] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.267876 23015 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:56.205969 23013 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:56.269312 22993 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:56.294701 20495 rpcz_store.cc:276] 0831 13:58:50.415848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:50.856545 (+440697us) service_pool.cc:225] Handling call
0831 13:58:50.856991 (+ 446us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:50.997534 (+140543us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:50.997561 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:51.002199 (+ 4638us) log.cc:844] Serialized 52 byte log entry
0831 13:58:51.360519 (+358320us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b4800080058. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c1838c41 00007fb9c182fde7 00007fb9c182fc7a 00007fb9c192506f 00007fb9c1924d37 00007fb9c193935c 00007fb9c1939308 00007fb9c1939288 00007fb9c1939243 00007fb9c19384ec 00007fb9c08b6656
0831 13:58:51.376365 (+ 15846us) spinlock_profiling.cc:243] Waited 9.36 ms on lock 0x7b18003b5c60. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171 00007fb9bc94909f
0831 13:58:51.532868 (+156503us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:51.549411 (+ 16543us) log.cc:844] Serialized 14 byte log entry
0831 13:58:51.559649 (+ 10238us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:51.793416 (+233767us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b540035c0d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:51.793457 (+ 41us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:56.241814 (+4448357us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:56.290044 (+ 48230us) raft_consensus.cc:1683] finished
0831 13:58:56.290075 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:56.290244 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40992,"prepare.queue_time_us":16364,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":10205,"raft.run_cpu_time_us":2849,"raft.run_wall_time_us":366092,"spinlock_wait_cycles":400810240,"thread_start_us":156304,"threads_started":1,"wal-append.queue_time_us":597559}
I20250831 13:58:56.300444 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.203960 22940 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:56.274204 21201 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:56.310384 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.287113 22543 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20250831 13:58:56.302024 22849 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.325721 23047 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:56.326143 22772 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:56.333463 20569 rpcz_store.cc:276] 0831 13:58:51.543759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.543875 (+ 116us) service_pool.cc:225] Handling call
0831 13:58:51.551601 (+ 7726us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:51.551898 (+ 297us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:51.551924 (+ 26us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:51.564112 (+ 12188us) log.cc:844] Serialized 26 byte log entry
0831 13:58:51.679669 (+115557us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:51.679708 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:51.679769 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:56.209243 (+4529474us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:56.271085 (+ 61842us) raft_consensus.cc:1683] finished
0831 13:58:56.271113 (+ 28us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:56.271325 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":24061,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":56997632,"thread_start_us":102838,"threads_started":1,"wal-append.queue_time_us":180507}
I20250831 13:58:56.204902 22839 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:58:56.332691 21204 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:56.338667 22923 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:56.336745 22991 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:56.200771 23027 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:56.338065 22861 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:56.359879 20642 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
W20250831 13:58:56.386089 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 85757.1us
W20250831 13:58:56.343912 22956 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: Append to log took a long time: real 3.247s user 0.007s sys 0.021s
I20250831 13:58:56.340570 22853 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.489009 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:56.493773 20541 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:56.505113 20575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1448) took 4484 ms (client timeout 1622 ms). Trace:
I20250831 13:58:56.507660 21209 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
W20250831 13:58:56.520241 23054 log.cc:927] Time spent T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20250831 13:58:56.361269 22849 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 1 FOLLOWER]: Advancing to term 2
W20250831 13:58:56.523250 20575 rpcz_store.cc:269] 0831 13:58:52.020182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:52.020327 (+ 145us) service_pool.cc:225] Handling call
0831 13:58:56.504856 (+4484529us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54001a77d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:56.505002 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":384210176}
W20250831 13:58:56.561504 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:56.653059 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:56.654462 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:56.659631 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:56.413997 23045 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:56.477767 23043 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:56.662374 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:56.408545 22839 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.667801 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:56.668594 20569 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:56.477018 23047 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:56.673183 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1428) took 4688 ms. Trace:
I20250831 13:58:56.673766 20501 rpcz_store.cc:276] 0831 13:58:51.984512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:51.984796 (+ 284us) service_pool.cc:225] Handling call
0831 13:58:52.003617 (+ 18821us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:52.044140 (+ 40523us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.672955 (+4628815us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b5400282b54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:56.673116 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91627904}
I20250831 13:58:56.661482 22940 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:56.525923 21201 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:56.671402 22839 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:56.673695 23047 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:58:56.696084 20614 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.021s sys 0.049s
I20250831 13:58:56.673269 23045 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:56.703326 20614 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.021s sys 0.050s
I20250831 13:58:56.691169 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1365) took 1431 ms. Trace:
I20250831 13:58:56.708583 20569 rpcz_store.cc:276] 0831 13:58:55.259963 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.673205 (+1413242us) service_pool.cc:225] Handling call
0831 13:58:56.690226 (+ 17021us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:56.690766 (+ 540us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:56.690805 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:56.690864 (+ 59us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:56.690910 (+ 46us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.690982 (+ 72us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:56.691106 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:56.684289 22940 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:56.703384 22899 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.026s sys 0.046s
W20250831 13:58:56.723266 22899 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s user 0.026s sys 0.046s
I20250831 13:58:56.708190 23045 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:56.680995 21175 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:56.725193 22940 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:56.740438 22853 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:56.744369 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1344) took 2337 ms. Trace:
I20250831 13:58:56.748592 20571 rpcz_store.cc:276] 0831 13:58:54.407244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:54.407389 (+ 145us) service_pool.cc:225] Handling call
0831 13:58:54.407742 (+ 353us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:54.987842 (+580100us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.744112 (+1756270us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54001fb154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:56.744308 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":339064704}
I20250831 13:58:56.749624 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1370) took 1488 ms. Trace:
I20250831 13:58:56.749887 20571 raft_consensus.cc:1238] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:56.749989 20569 rpcz_store.cc:276] 0831 13:58:55.261429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.712307 (+1450878us) service_pool.cc:225] Handling call
0831 13:58:56.749159 (+ 36852us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:56.749336 (+ 177us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:56.749359 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:56.749397 (+ 38us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:56.749427 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.749489 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:56.749577 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:56.750533 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1371) took 1488 ms. Trace:
I20250831 13:58:56.750921 20571 rpcz_store.cc:276] 0831 13:58:55.261710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.749474 (+1487764us) service_pool.cc:225] Handling call
0831 13:58:56.749733 (+ 259us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:56.750324 (+ 591us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.750486 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:56.752579 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1378) took 1381 ms. Trace:
I20250831 13:58:56.752988 20571 rpcz_store.cc:276] 0831 13:58:55.371306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.751716 (+1380410us) service_pool.cc:225] Handling call
0831 13:58:56.752010 (+ 294us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:56.752213 (+ 203us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:56.752245 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:56.752292 (+ 47us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:56.752340 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.752409 (+ 69us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:56.752521 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:56.746090 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1255) took 4549 ms. Trace:
I20250831 13:58:56.702821 22605 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:56.747421 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:56.758262 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.732371 22881 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:56.761395 20642 raft_consensus.cc:2391] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 8: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:58:56.767846 22920 log.cc:927] Time spent T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Append to log took a long time: real 4.978s user 0.010s sys 0.028s
I20250831 13:58:56.754884 20648 rpcz_store.cc:276] 0831 13:58:52.196625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:52.198760 (+ 2135us) service_pool.cc:225] Handling call
0831 13:58:52.199074 (+ 314us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:52.292593 (+ 93519us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:52.292635 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:52.293188 (+ 553us) log.cc:844] Serialized 26 byte log entry
0831 13:58:53.135818 (+842630us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:53.182053 (+ 46235us) log.cc:844] Serialized 14 byte log entry
0831 13:58:53.195131 (+ 13078us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.745671 (+3550540us) spinlock_profiling.cc:243] Waited 82.3 ms on lock 0x7b5400283554. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:56.745726 (+ 55us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:56.745782 (+ 56us) raft_consensus.cc:1683] finished
0831 13:58:56.745875 (+ 93us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:56.746010 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23814,"prepare.queue_time_us":114346,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":44417,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":34424,"spinlock_wait_cycles":189292416,"thread_start_us":818877,"threads_started":2,"wal-append.queue_time_us":782181,"wal-append.run_cpu_time_us":2747,"wal-append.run_wall_time_us":1112279}
I20250831 13:58:56.760468 23049 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:56.758555 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1382) took 1377 ms. Trace:
I20250831 13:58:56.764963 22327 raft_consensus.cc:3053] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:56.781122 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.783634 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:56.764170 23046 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:56.785950 20648 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:58:56.771021 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 97747.6us
I20250831 13:58:56.790436 20541 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:56.733820 22794 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.800691 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:58:56.779974 22327 raft_consensus.cc:738] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:56.779421 20571 rpcz_store.cc:276] 0831 13:58:55.381554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.754021 (+1372467us) service_pool.cc:225] Handling call
0831 13:58:56.754331 (+ 310us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:56.754531 (+ 200us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:56.754566 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:56.754618 (+ 52us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:56.754665 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.758420 (+ 3755us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:56.758512 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:56.807909 22773 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:56.797943 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:56.785871 22905 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:58:56.809667 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.008s sys 0.009s
W20250831 13:58:56.903633 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s user 0.008s sys 0.009s
W20250831 13:58:56.811251 22938 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.022s sys 0.073s
I20250831 13:58:56.803963 22881 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:58:56.822957 20468 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.009s sys 0.012s
W20250831 13:58:56.818293 22347 consensus_meta.cc:287] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.524s user 0.013s sys 0.000s
W20250831 13:58:56.826639 20467 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.015s sys 0.032s
W20250831 13:58:56.915216 20467 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s user 0.015s sys 0.032s
I20250831 13:58:56.826928 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:56.827589 22808 consensus_meta.cc:287] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.587s user 0.010s sys 0.000s
W20250831 13:58:56.829416 21205 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.031s sys 0.061s
I20250831 13:58:56.846939 22605 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:56.833647 23038 consensus_meta.cc:287] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.594s user 0.010s sys 0.001s
W20250831 13:58:56.865309 22986 consensus_meta.cc:287] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.599s user 0.008s sys 0.001s
I20250831 13:58:56.864145 22794 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:56.895874 20643 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s user 0.052s sys 0.075s
I20250831 13:58:56.901584 20642 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:56.908586 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:56.929088 20617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.002s sys 0.022s
W20250831 13:58:56.864749 23055 log.cc:927] Time spent T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
W20250831 13:58:56.913676 20468 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s user 0.010s sys 0.012s
W20250831 13:58:56.931710 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 673007us
W20250831 13:58:56.915769 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 638213us
W20250831 13:58:56.910171 22968 consensus_meta.cc:287] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.566s user 0.009s sys 0.000s
I20250831 13:58:56.832366 22327 consensus_queue.cc:260] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.952939 22327 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:56.953822 23056 log.cc:927] Time spent T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20250831 13:58:56.911823 22938 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s user 0.022s sys 0.073s
W20250831 13:58:56.929418 20617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s user 0.002s sys 0.022s
W20250831 13:58:56.921840 21205 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.719s user 0.031s sys 0.061s
W20250831 13:58:56.929715 20643 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s user 0.052s sys 0.075s
I20250831 13:58:56.946549 22977 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:56.881378 23027 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.935196 20467 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:56.948642 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:56.944483 22338 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:56.970887 22338 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.945886 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:56.974061 20495 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:56.975298 20642 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:58:56.975389 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:56.976670 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:56.976902 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:56.977725 22347 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:56.977892 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:56.978279 22347 raft_consensus.cc:695] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:56.979053 20495 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:56.979193 22347 consensus_queue.cc:237] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.979416 22338 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:56.981630 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:56.924029 22808 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:56.986672 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 51631.7us
I20250831 13:58:56.929662 22986 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.929486 23038 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.989144 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:56.946363 22968 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.994470 22898 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:57.004302 22898 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:56.995271 22541 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:56.995350 23057 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:57.005743 23057 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.008323 23057 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:56.995802 22338 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:57.008994 22993 consensus_meta.cc:287] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.533s user 0.010s sys 0.000s
I20250831 13:58:57.009225 22338 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.009425 22993 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.012514 22338 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:57.013311 22986 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.019865 20495 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:57.021409 20495 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:56.983135 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.004904 22938 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:56.997107 23046 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 13:58:57.040726 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.053596 22899 consensus_queue.cc:1046] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:57.054431 23038 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.054819 23057 raft_consensus.cc:3053] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:57.055166 23057 raft_consensus.cc:738] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:57.055931 23057 consensus_queue.cc:260] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.040488 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.057879 21207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47742 (request call id 99) took 1408 ms. Trace:
I20250831 13:58:57.058275 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.058338 21207 rpcz_store.cc:276] 0831 13:58:55.856249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.906788 (+ 50539us) service_pool.cc:225] Handling call
0831 13:58:57.057111 (+1150323us) spinlock_profiling.cc:243] Waited 967 us on lock 0x7b58000c0098. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:57.057336 (+ 225us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.057531 (+ 195us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:57.057561 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:57.057614 (+ 53us) raft_consensus.cc:1643] Marking committed up to 7
0831 13:58:57.057653 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.057719 (+ 66us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.057822 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2224000}
I20250831 13:58:57.059581 23057 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:57.049701 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 56335.1us
I20250831 13:58:57.056222 23075 consensus_queue.cc:1046] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 13:58:57.072283 20617 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:57.075956 22965 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.076484 22965 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.078539 22965 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.091295 20617 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:57.066720 23070 log.cc:927] Time spent T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20250831 13:58:57.065311 23046 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:57.041671 22986 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:58:57.098042 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:56.988287 22977 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.104072 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1434) took 4976 ms. Trace:
W20250831 13:58:57.083958 23053 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250831 13:58:57.069172 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47732 (request call id 178) took 1641 ms. Trace:
I20250831 13:58:57.085728 22338 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:57.100666 23058 raft_consensus.cc:2802] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:58:57.105933 23046 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:57.104295 22815 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:58:57.085309 22849 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.103806 22951 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.106321 22949 consensus_queue.cc:1046] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20250831 13:58:57.106986 22986 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:58:57.107061 20500 rpcz_store.cc:276] 0831 13:58:52.127732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:52.131049 (+ 3317us) service_pool.cc:225] Handling call
0831 13:58:52.131308 (+ 259us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:52.131514 (+ 206us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:58:56.780226 (+4648712us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b1800044500. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9bc5c255c 00007fb9b99f777b 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a11805 00007fb9c035b327 00007fb9bc980957 00007fb9bc99006f
0831 13:58:56.780421 (+ 195us) log.cc:844] Serialized 14 byte log entry
0831 13:58:57.103449 (+323028us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:57.103609 (+ 160us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:57.103679 (+ 70us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.103892 (+ 213us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.104013 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6152,"prepare.queue_time_us":104453,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":51150,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":68596992,"thread_start_us":465061,"threads_started":3,"wal-append.queue_time_us":213505}
W20250831 13:58:57.236570 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:57.226985 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 196174us
I20250831 13:58:57.134661 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1307) took 1059 ms. Trace:
I20250831 13:58:57.246235 20648 rpcz_store.cc:276] 0831 13:58:56.075690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.920135 (+844445us) service_pool.cc:225] Handling call
0831 13:58:56.920441 (+ 306us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:56.920731 (+ 290us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:56.920748 (+ 17us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:56.921073 (+ 325us) log.cc:844] Serialized 26 byte log entry
0831 13:58:56.985215 (+ 64142us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:56.985277 (+ 62us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:56.985385 (+ 108us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:57.134419 (+149034us) raft_consensus.cc:1683] finished
0831 13:58:57.134442 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.134598 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51,"raft.queue_time_us":82,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":3424,"thread_start_us":63696,"threads_started":1,"wal-append.queue_time_us":84915}
I20250831 13:58:57.248453 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:57.227527 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 214963us
W20250831 13:58:57.251168 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.229791 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.250478 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.295845 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.297765 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:57.298960 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 360585us
I20250831 13:58:57.300146 20616 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.301560 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.303020 20500 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.311199 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.311659 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.312629 20500 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:57.314844 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1361) took 2283 ms. Trace:
I20250831 13:58:57.323341 20574 rpcz_store.cc:276] 0831 13:58:55.031269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.875074 (+843805us) service_pool.cc:225] Handling call
0831 13:58:57.314330 (+1439256us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54000b3bd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.314609 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":479164288}
W20250831 13:58:57.324404 20576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1350) took 2814 ms (client timeout 3288 ms). Trace:
I20250831 13:58:57.308013 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1461) took 1705 ms. Trace:
W20250831 13:58:57.324954 20576 rpcz_store.cc:269] 0831 13:58:54.509998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.010224 (+500226us) service_pool.cc:225] Handling call
0831 13:58:57.323981 (+2313757us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002ff654. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.324209 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2583159168}
I20250831 13:58:57.325300 20498 rpcz_store.cc:276] 0831 13:58:55.602631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.602814 (+ 183us) service_pool.cc:225] Handling call
0831 13:58:55.639744 (+ 36930us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:56.038718 (+398974us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:56.038764 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:56.038822 (+ 58us) raft_consensus.cc:1643] Marking committed up to 7
0831 13:58:56.038867 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.307473 (+1268606us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b5400011dd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:57.307537 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.307804 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":916977024}
I20250831 13:58:57.302846 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:57.314720 20616 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.321642 20500 raft_consensus.cc:1273] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20250831 13:58:57.327817 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:57.328052 20573 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34436 (request call id 62) took 3012 ms (client timeout 1521 ms). Trace:
W20250831 13:58:57.328605 20573 rpcz_store.cc:269] 0831 13:58:54.315739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:54.315870 (+ 131us) service_pool.cc:225] Handling call
0831 13:58:57.327560 (+3011690us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b540031e2d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.327806 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1543179136}
I20250831 13:58:57.315165 20493 raft_consensus.cc:1273] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:57.331792 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1376) took 2015 ms. Trace:
I20250831 13:58:57.332271 20569 rpcz_store.cc:276] 0831 13:58:55.315175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.751010 (+1435835us) service_pool.cc:225] Handling call
0831 13:58:57.328498 (+577488us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.329141 (+ 643us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:57.329168 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:57.329230 (+ 62us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:57.329281 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.329358 (+ 77us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.330301 (+ 943us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b54002ff650. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.330506 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1261081856}
I20250831 13:58:57.333088 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:57.334074 20649 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 1004) took 3271 ms (client timeout 1545 ms). Trace:
W20250831 13:58:57.334174 21203 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47746 (request call id 81) took 4165 ms (client timeout 1847 ms). Trace:
W20250831 13:58:57.335932 20467 connection.cc:232] RPC call timeout handler was delayed by -1.4353751370000509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20250831 13:58:57.336485 20467 connection.cc:232] RPC call timeout handler was delayed by -1.3597751370000424s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16720000000000002s for a response
I20250831 13:58:57.326534 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:54.312682 20502 rpcz_store.cc:276] 0831 13:58:52.184030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.152729 (+968699us) service_pool.cc:225] Handling call
0831 13:58:53.153068 (+ 339us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:53.155305 (+ 2237us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:53.155323 (+ 18us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:53.155626 (+ 303us) log.cc:844] Serialized 26 byte log entry
0831 13:58:53.283855 (+128229us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:54.311625 (+1027770us) log.cc:844] Serialized 14 byte log entry
0831 13:58:54.311926 (+ 301us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.311994 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:54.312031 (+ 37us) raft_consensus.cc:1683] finished
0831 13:58:54.312043 (+ 12us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:54.312152 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1853,"prepare.queue_time_us":192730,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":5958,"spinlock_wait_cycles":10062208,"thread_start_us":1082321,"threads_started":2,"wal-append.queue_time_us":171028}
W20250831 13:58:57.340862 20494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1432) took 5213 ms (client timeout 1654 ms). Trace:
I20250831 13:58:57.341547 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.345279 20496 raft_consensus.cc:1273] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250831 13:58:57.334548 21203 rpcz_store.cc:269] 0831 13:58:53.168992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.182135 (+ 13143us) service_pool.cc:225] Handling call
0831 13:58:57.302385 (+4120250us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b58000c0098. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d75ce 000000000035a3c9 00007fb9c185e300 00007fb9c1924ec4 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:57.334049 (+ 31664us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25551104}
W20250831 13:58:57.352185 20645 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 1006) took 3086 ms (client timeout 1966 ms). Trace:
W20250831 13:58:57.352612 20577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1349) took 2845 ms (client timeout 1672 ms). Trace:
W20250831 13:58:57.353124 20577 rpcz_store.cc:269] 0831 13:58:54.506832 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:54.506997 (+ 165us) service_pool.cc:225] Handling call
0831 13:58:57.352193 (+2845196us) spinlock_profiling.cc:243] Waited 46.9 ms on lock 0x7b54000b54d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.352415 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":107852160}
I20250831 13:58:57.354315 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1283) took 2829 ms. Trace:
I20250831 13:58:57.354775 20644 rpcz_store.cc:276] 0831 13:58:54.524969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.428352 (+903383us) service_pool.cc:225] Handling call
0831 13:58:55.498303 (+ 69951us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:55.703110 (+204807us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:55.703141 (+ 31us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:55.703184 (+ 43us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:55.703219 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.353818 (+1650599us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54000307d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:57.353873 (+ 55us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.354120 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":470547712}
I20250831 13:58:57.165381 21206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46230 (request call id 2) took 4807 ms. Trace:
I20250831 13:58:57.358754 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.359234 21206 rpcz_store.cc:276] 0831 13:58:52.488651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:52.504409 (+ 15758us) service_pool.cc:225] Handling call
0831 13:58:57.163440 (+4659031us) spinlock_profiling.cc:243] Waited 28 ms on lock 0x7b540040e320. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc90ac50 00007fb9bc97caf2 00007fb9bc97b7b0 00007fb9bc9749ba 00007fb9c18d4079 00007fb9baf64644 00007fb9baf645d2 00007fb9baf64501 00007fb9baf6447c 00007fb9baf63732
0831 13:58:57.165010 (+ 1570us) spinlock_profiling.cc:243] Waited 3.58 s on lock 0x7b540040e054. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97c6e5 00007fb9bc9749ba 00007fb9c18d4079 00007fb9baf64644 00007fb9baf645d2 00007fb9baf64501 00007fb9baf6447c 00007fb9baf63732 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:57.165264 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8309432576}
I20250831 13:58:57.360193 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.361536 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:57.362529 20447 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 90a0a56228d04660b3cb351485be6593 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20250831 13:58:57.128327 21205 rpcz_store.cc:276] 0831 13:58:55.427865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.431540 (+ 3675us) service_pool.cc:225] Handling call
0831 13:58:55.560488 (+128948us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:56.004290 (+443802us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:56.009007 (+ 4717us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:56.067675 (+ 58668us) log.cc:844] Serialized 26 byte log entry
0831 13:58:56.166782 (+ 99107us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:56.989481 (+822699us) log.cc:844] Serialized 14 byte log entry
0831 13:58:57.003753 (+ 14272us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.015691 (+ 11938us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:57.030439 (+ 14748us) raft_consensus.cc:1683] finished
0831 13:58:57.042823 (+ 12384us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.067705 (+ 24882us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":712956,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":428912,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":10575,"thread_start_us":1130601,"threads_started":3,"wal-append.queue_time_us":165047}
I20250831 13:58:57.336400 20650 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:57.365173 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1034) took 1187 ms. Trace:
I20250831 13:58:57.365661 20650 rpcz_store.cc:276] 0831 13:58:56.177818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.335570 (+1157752us) service_pool.cc:225] Handling call
0831 13:58:57.335962 (+ 392us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.364772 (+ 28810us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.365041 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:57.367753 20650 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:57.368680 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1036) took 1172 ms. Trace:
I20250831 13:58:57.369308 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1298) took 1304 ms. Trace:
W20250831 13:58:57.341302 20494 rpcz_store.cc:269] 0831 13:58:52.126997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:52.130411 (+ 3414us) service_pool.cc:225] Handling call
0831 13:58:57.340390 (+5209979us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b54001293d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.340625 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5156230400}
W20250831 13:58:57.371255 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:57.371745 20568 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1473) took 1427 ms (client timeout 1725 ms). Trace:
I20250831 13:58:57.372252 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1308) took 1286 ms. Trace:
I20250831 13:58:57.372445 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:57.373425 20571 raft_consensus.cc:1238] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20250831 13:58:57.373716 20569 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:57.373981 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.375108 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34436 (request call id 64) took 1670 ms. Trace:
I20250831 13:58:57.375402 20644 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:57.375619 20571 rpcz_store.cc:276] 0831 13:58:55.704230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.902025 (+1197795us) service_pool.cc:225] Handling call
0831 13:58:57.372711 (+470686us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:57.373978 (+ 1267us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.374836 (+ 858us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54001a72d0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.375013 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1081325312}
I20250831 13:58:57.377152 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:57.379032 20573 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:57.378935 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.380653 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.367136 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1304) took 1300 ms. Trace:
I20250831 13:58:57.376874 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:57.372215 20568 rpcz_store.cc:269] 0831 13:58:55.944049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.961578 (+ 17529us) service_pool.cc:225] Handling call
0831 13:58:57.371327 (+1409749us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b54001a72d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.371527 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":104683264}
I20250831 13:58:57.373104 20642 rpcz_store.cc:276] 0831 13:58:56.085885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.074503 (+988618us) service_pool.cc:225] Handling call
0831 13:58:57.074898 (+ 395us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.131193 (+ 56295us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:57.131229 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:57.131366 (+ 137us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:57.131401 (+ 35us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.371657 (+240256us) spinlock_profiling.cc:243] Waited 55.9 ms on lock 0x7b54001ef854. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:57.371746 (+ 89us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.372020 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":128675584}
I20250831 13:58:57.369911 20646 rpcz_store.cc:276] 0831 13:58:56.064609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.064736 (+ 127us) service_pool.cc:225] Handling call
0831 13:58:57.351974 (+1287238us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.352535 (+ 561us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:57.352559 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:57.352625 (+ 66us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:57.352672 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.352758 (+ 86us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.368878 (+ 16120us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400069d50. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.369050 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2959540480}
I20250831 13:58:57.376180 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1035) took 1198 ms. Trace:
W20250831 13:58:57.352672 20645 rpcz_store.cc:269] 0831 13:58:54.265374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:54.324222 (+ 58848us) service_pool.cc:225] Handling call
0831 13:58:57.351732 (+3027510us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b5400069d54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.351970 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1246466432}
I20250831 13:58:57.388669 20644 rpcz_store.cc:276] 0831 13:58:56.178167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.362490 (+1184323us) service_pool.cc:225] Handling call
0831 13:58:57.374966 (+ 12476us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.375912 (+ 946us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.376106 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:57.376361 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:57.334479 20649 rpcz_store.cc:269] 0831 13:58:54.062405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:54.062569 (+ 164us) service_pool.cc:225] Handling call
0831 13:58:57.332737 (+3270168us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:58:57.333977 (+ 1240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2952064}
I20250831 13:58:57.128926 22338 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:58:57.404875 20646 raft_consensus.cc:1238] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
I20250831 13:58:57.404644 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:57.405805 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1038) took 1202 ms. Trace:
I20250831 13:58:57.405896 20576 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:57.407234 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:57.411186 20496 raft_consensus.cc:1238] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:57.414862 20645 raft_consensus.cc:1238] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 4. Current term is 5. Ops: []
W20250831 13:58:57.417198 20570 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1354) took 2879 ms (client timeout 1588 ms). Trace:
I20250831 13:58:57.417593 22100 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:57.404805 22338 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:57.369110 20650 rpcz_store.cc:276] 0831 13:58:56.196621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.366798 (+1170177us) service_pool.cc:225] Handling call
0831 13:58:57.367342 (+ 544us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.368325 (+ 983us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.368586 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:57.406270 20646 rpcz_store.cc:276] 0831 13:58:56.203675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.387895 (+1184220us) service_pool.cc:225] Handling call
0831 13:58:57.404537 (+ 16642us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.405494 (+ 957us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.405735 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:57.417717 20570 rpcz_store.cc:269] 0831 13:58:54.537343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:54.855125 (+317782us) service_pool.cc:225] Handling call
0831 13:58:57.382929 (+2527804us) spinlock_profiling.cc:243] Waited 93 ms on lock 0x7b54001fa754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.416980 (+ 34051us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":213838592}
W20250831 13:58:57.157966 23037 consensus_meta.cc:287] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.843s user 0.009s sys 0.000s
I20250831 13:58:57.144829 22940 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.431574 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:46230 (request call id 3) took 2396 ms. Trace:
I20250831 13:58:57.383183 20643 rpcz_store.cc:276] 0831 13:58:56.066838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.113212 (+ 46374us) service_pool.cc:225] Handling call
0831 13:58:56.114325 (+ 1113us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:56.147579 (+ 33254us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:56.147609 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:56.148021 (+ 412us) log.cc:844] Serialized 26 byte log entry
0831 13:58:57.365043 (+1217022us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b1800471700. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9bc5c255c 00007fb9b99f777b 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171
0831 13:58:57.365457 (+ 414us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:57.366015 (+ 558us) log.cc:844] Serialized 14 byte log entry
0831 13:58:57.366361 (+ 346us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.366478 (+ 117us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:57.366589 (+ 111us) raft_consensus.cc:1683] finished
0831 13:58:57.366719 (+ 130us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.366917 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51548,"raft.queue_time_us":35178,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":2621696,"thread_start_us":799825,"threads_started":1,"wal-append.queue_time_us":808649}
I20250831 13:58:57.431384 23037 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.432009 21208 rpcz_store.cc:276] 0831 13:58:55.035059 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.035223 (+ 164us) service_pool.cc:225] Handling call
0831 13:58:57.384584 (+2349361us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540040e054. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8facd0 00007fb9bc97eeea 00007fb9c18d760e 00007fb9baf74cc7 00007fb9baf74c52 00007fb9baf74b81 00007fb9baf74afc 00007fb9baf73db2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.431266 (+ 46682us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2424634752}
I20250831 13:58:57.148108 22968 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.436919 23037 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.142285 22986 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:58:57.176530 22977 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:57.438416 20575 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:57.140395 22951 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.442111 20574 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:58:57.441552 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1037) took 1240 ms. Trace:
I20250831 13:58:57.163662 22874 raft_consensus.cc:3053] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:57.180023 22897 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:57.442838 20642 rpcz_store.cc:276] 0831 13:58:56.199776 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.386898 (+1187122us) service_pool.cc:225] Handling call
0831 13:58:57.403147 (+ 16249us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.439412 (+ 36265us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:57.439463 (+ 51us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:57.439547 (+ 84us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:57.439604 (+ 57us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.439669 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.439803 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:57.170627 22849 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.185578 22979 raft_consensus.cc:3053] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:57.444887 22979 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:58:57.150962 23015 consensus_meta.cc:287] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.789s user 0.006s sys 0.005s
I20250831 13:58:57.518078 20494 raft_consensus.cc:2391] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.519301 23015 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.436290 22940 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:57.111975 23022 consensus_meta.cc:287] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.872s user 0.010s sys 0.000s
I20250831 13:58:57.443313 22874 raft_consensus.cc:738] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:58:57.150987 23047 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:57.140354 22987 consensus_meta.cc:287] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.827s user 0.011s sys 0.000s
W20250831 13:58:57.185410 22923 consensus_meta.cc:287] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.680s user 0.007s sys 0.003s
W20250831 13:58:57.142910 22528 consensus_meta.cc:287] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.800s user 0.002s sys 0.008s
W20250831 13:58:57.163225 22872 consensus_meta.cc:287] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.820s user 0.008s sys 0.003s
W20250831 13:58:57.139619 22999 consensus_meta.cc:287] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.794s user 0.011s sys 0.000s
W20250831 13:58:57.154228 22780 consensus_meta.cc:287] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.863s user 0.009s sys 0.000s
W20250831 13:58:57.143594 23034 log.cc:927] Time spent T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Append to log took a long time: real 1.115s user 0.003s sys 0.013s
I20250831 13:58:57.532930 22901 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:57.534418 22543 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:57.534991 22852 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.566972 22852 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.568600 22758 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.569379 22852 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.535285 22429 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:57.535936 22975 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:57.571157 22975 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:57.572250 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.573696 22975 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:57.574344 22873 consensus_queue.cc:1046] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:58:57.563511 22901 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:57.578948 22901 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:57.570605 22429 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.582372 22429 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.539752 22922 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.585834 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.543795 22998 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:57.586201 22922 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.587107 22998 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:57.588830 22922 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.125180 22993 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:57.148047 23024 log.cc:927] Time spent T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602: Append to log took a long time: real 1.138s user 0.002s sys 0.009s
W20250831 13:58:57.551806 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:57.186061 22861 consensus_meta.cc:287] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.532s user 0.010s sys 0.000s
I20250831 13:58:57.159086 23027 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.137940 23058 raft_consensus.cc:695] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:57.563110 20493 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.564550 22614 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.556005 22819 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:57.599084 22819 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.601549 22819 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.601224 22979 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.602648 22979 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:57.598551 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.187714 22966 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.585196 22959 raft_consensus.cc:3053] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:57.585947 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.539666 23019 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:57.545810 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.620337 20494 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 10.
I20250831 13:58:57.545325 22794 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:57.564962 23047 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.551807 23079 raft_consensus.cc:3053] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:57.602649 22614 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.143952 22808 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.153858 22974 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.612310 22872 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.612772 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.614214 22987 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:57.615816 22998 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.615926 23015 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.198264 23046 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.617515 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.617883 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1296) took 2621 ms. Trace:
I20250831 13:58:57.145375 22839 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.618009 22528 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.147056 22898 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:57.626086 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.627475 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.617766 22687 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:57.639011 20617 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:57.610918 22959 raft_consensus.cc:738] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:57.642848 22872 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.623000 23079 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:57.643580 22959 consensus_queue.cc:260] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.631150 22879 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.639009 22861 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.618039 22923 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:57.603729 23058 consensus_queue.cc:237] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.639709 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.631771 22614 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.644033 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.656193 22861 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.656555 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.138648 22815 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:57.632951 20647 rpcz_store.cc:276] 0831 13:58:54.995894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.524027 (+528133us) service_pool.cc:225] Handling call
0831 13:58:55.524403 (+ 376us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:55.592507 (+ 68104us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:55.592539 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:55.592978 (+ 439us) log.cc:844] Serialized 26 byte log entry
0831 13:58:55.710978 (+118000us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:55.711020 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:55.711086 (+ 66us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:57.142993 (+1431907us) spinlock_profiling.cc:243] Waited 23.3 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:58:57.617404 (+474411us) raft_consensus.cc:1683] finished
0831 13:58:57.617447 (+ 43us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.617702 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":276666,"raft.queue_time_us":96160,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":18828,"spinlock_wait_cycles":53536640,"thread_start_us":117709,"threads_started":1,"wal-append.queue_time_us":366141}
I20250831 13:58:57.657837 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.644955 22959 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:57.652199 22372 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:57.661764 20499 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 9. Current term is 10. Ops: []
I20250831 13:58:57.662087 22372 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:57.662611 22372 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:57.183507 23045 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:57.627246 23019 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:57.667959 23028 log.cc:927] Time spent T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 1.699s user 0.002s sys 0.012s
I20250831 13:58:57.668390 23019 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:57.644407 23079 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:57.670099 22931 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:57.670349 23079 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:57.655680 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.672129 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.675076 20494 raft_consensus.cc:2391] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.676596 20500 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:57.670699 22931 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.682366 20572 raft_consensus.cc:2408] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:57.683104 20502 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:57.625859 22794 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.690922 23001 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.696681 22815 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:57.691218 22592 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:57.688349 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.695696 22799 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.016s
I20250831 13:58:57.692890 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.691125 22951 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.631140 22874 consensus_queue.cc:260] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:57.686805 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.709528 20498 raft_consensus.cc:2391] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 2: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.715436 20501 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:57.696725 23001 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.718030 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:57.724844 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.726544 22931 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.726805 22999 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.729601 22999 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.730923 22976 consensus_queue.cc:1035] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20250831 13:58:57.731850 22874 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:57.732120 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:57.733932 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.734095 23001 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.734638 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:57.725270 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:57.736266 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 95400.7us
I20250831 13:58:57.742250 20498 raft_consensus.cc:2366] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:58:57.748706 20570 raft_consensus.cc:2391] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:57.750080 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.750206 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.751653 20495 raft_consensus.cc:2391] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:57.719415 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 56461.4us
I20250831 13:58:57.752552 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.756235 20494 raft_consensus.cc:2391] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.755405 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.754632 20651 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:57.759128 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.707366 22780 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.763079 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.763091 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:57.761747 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.764345 22359 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:57.764849 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:58:57.764902 22359 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.766665 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.768800 23045 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:57.769706 22359 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:57.769973 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37606 (request call id 61) took 5585 ms. Trace:
W20250831 13:58:57.770910 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.767726 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.778558 22780 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.779002 20568 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 9. Current term is 10. Ops: []
I20250831 13:58:57.780800 23046 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.782117 22839 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:58:57.782254 20572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1377) took 2422 ms (client timeout 1992 ms). Trace:
W20250831 13:58:57.782125 20542 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41400 dropped due to backpressure: service queue is full with 50 items
I20250831 13:58:57.782665 22372 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:57.782759 20572 rpcz_store.cc:269] 0831 13:58:55.360285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:55.607071 (+246786us) service_pool.cc:225] Handling call
0831 13:58:57.782039 (+2174968us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b540031ddd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:57.782185 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4525538048}
I20250831 13:58:57.783460 22794 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.786712 20643 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:57.789350 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.772316 20497 rpcz_store.cc:276] 0831 13:58:52.184129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:53.152735 (+968606us) service_pool.cc:225] Handling call
0831 13:58:53.156274 (+ 3539us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:53.161828 (+ 5554us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:53.161873 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:53.181418 (+ 19545us) log.cc:844] Serialized 52 byte log entry
0831 13:58:53.850564 (+669146us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:54.315561 (+464997us) log.cc:844] Serialized 14 byte log entry
0831 13:58:54.315755 (+ 194us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:54.315822 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:57.682498 (+3366676us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b1800259540. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b9a14792 00007fb9b9a16b61 00007fb9b9a16b19 00007fb9b9a16aa9 00007fb9b9a16a71 00007fb9b9a15d6d 0000000000379254 0000000000379099 00007fb9b99f7b1a
0831 13:58:57.769526 (+ 87028us) raft_consensus.cc:1683] finished
0831 13:58:57.769613 (+ 87us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.769793 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":444585,"prepare.queue_time_us":520254,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":7758,"raft.queue_time_us":11419,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":12656,"spinlock_wait_cycles":15856000,"thread_start_us":644121,"threads_started":1,"wal-append.queue_time_us":1210437}
I20250831 13:58:57.791106 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.792445 20643 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:57.792225 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.792966 20573 raft_consensus.cc:1238] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:57.792827 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:57.793220 20495 raft_consensus.cc:2391] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.793812 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:57.794116 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.801847 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.795078 20498 raft_consensus.cc:2391] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.795199 22976 consensus_queue.cc:1046] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:57.712504 23022 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.797552 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.807222 20493 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:57.798554 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:57.794531 22999 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.804770 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.805562 20494 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:57.812175 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.813290 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.813834 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1462) took 1956 ms. Trace:
I20250831 13:58:57.814164 20573 rpcz_store.cc:276] 0831 13:58:55.853223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.792335 (+1939112us) service_pool.cc:225] Handling call
0831 13:58:57.792688 (+ 353us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.807196 (+ 14508us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.807389 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:57.814324 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.810740 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.820750 22338 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.821614 22369 raft_consensus.cc:3053] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:57.822149 22369 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:57.809829 22780 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:57.826094 22780 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.809476 22372 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:57.827947 22780 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.828150 23022 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.828773 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:57.829734 20543 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.831068 20648 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:57.831732 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:57.832248 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1032) took 1684 ms. Trace:
I20250831 13:58:57.832736 23045 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:57.832659 20648 rpcz_store.cc:276] 0831 13:58:56.148157 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.301139 (+1152982us) service_pool.cc:225] Handling call
0831 13:58:57.301654 (+ 515us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.831443 (+529789us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.832047 (+ 604us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54002ea254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:57.832178 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1217150720}
I20250831 13:58:57.833254 20543 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:57.833530 23045 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:57.834326 23045 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250831 13:58:57.834662 20543 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.817608 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.840353 21174 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.845293 22924 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:58:57.846405 23088 log.cc:927] Time spent T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20250831 13:58:57.852275 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.851428 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.851218 22369 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.853644 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.859484 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.860184 20494 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:57.857693 20466 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:57.834777 23045 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:57.864111 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 56728.1us
I20250831 13:58:57.855037 20570 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:57.840993 22338 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:57.871176 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 127180us
I20250831 13:58:57.869992 22780 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:57.871834 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.872036 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.872445 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.872640 20499 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:57.872789 20543 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.873543 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.874045 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.865372 23045 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:58:57.875649 20495 raft_consensus.cc:2391] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.876916 20643 raft_consensus.cc:1273] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:57.872666 22369 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:58:57.881259 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.882176 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.901872 22896 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:57.902415 22896 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:57.902832 22896 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:57.907089 20501 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:57.907331 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.908564 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.909153 20495 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:57.909988 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.911178 22882 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:57.911492 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46464 (request call id 82) took 1184 ms. Trace:
I20250831 13:58:57.911592 22882 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.912066 22882 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:57.911939 20644 rpcz_store.cc:276] 0831 13:58:56.727367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.424272 (+696905us) service_pool.cc:225] Handling call
0831 13:58:57.533523 (+109251us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.661465 (+127942us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:57.661497 (+ 32us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:57.661573 (+ 76us) raft_consensus.cc:1643] Marking committed up to 7
0831 13:58:57.661615 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.911286 (+249671us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54003a11d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:57.911329 (+ 43us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:57.911442 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":286093568}
I20250831 13:58:57.913980 20501 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:57.871326 22900 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:57.915328 22900 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.916548 22905 consensus_queue.cc:1046] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250831 13:58:57.918866 22839 raft_consensus.cc:3053] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:57.919301 22839 raft_consensus.cc:738] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:57.920055 22799 raft_consensus.cc:3053] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:57.910473 20493 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:57.921444 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:57.920450 22839 consensus_queue.cc:260] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.890311 22528 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.871332 22833 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:57.931015 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.783210 20542 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41400 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1397)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34436 (request call id 67)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1505)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1503)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1501)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34436 (request call id 66)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1403)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1465)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1466)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1467)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1468)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1469)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1470)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1472)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1474)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1476)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1477)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1481)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1493)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1495)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1499)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1502)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1395)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1396)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1402)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1404)
I20250831 13:58:57.920483 22799 raft_consensus.cc:738] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:57.878947 22999 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.935717 22799 consensus_queue.cc:260] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.936676 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:57.937060 22799 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:57.937479 22999 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.937264 22896 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.892627 22780 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 13:58:57.938975 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 67092us
I20250831 13:58:57.939514 22896 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:57.935895 20496 raft_consensus.cc:2391] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 8: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:57.940701 20616 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:57.940857 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.942525 20496 raft_consensus.cc:1238] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:57.942983 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:57.943413 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 70775.6us
I20250831 13:58:57.943686 20576 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:57.943804 20643 raft_consensus.cc:1238] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:57.944129 20543 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:57.938998 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.945734 20495 raft_consensus.cc:2391] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 8: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:57.946741 20495 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:57.946974 20617 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:58:57.947451 22837 consensus_queue.cc:1046] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:58:57.944140 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.948544 23058 raft_consensus.cc:3053] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Stepping down as leader of term 9
I20250831 13:58:57.948380 22837 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:58:57.948911 23058 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:57.949524 22837 raft_consensus.cc:695] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:57.950173 23058 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.950681 22837 consensus_queue.cc:237] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.951038 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:58:57.951877 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:57.952268 22839 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:57.943800 20496 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:57.953672 20496 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:57.953976 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1363) took 2856 ms. Trace:
I20250831 13:58:57.954334 20575 rpcz_store.cc:276] 0831 13:58:55.097888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.656002 (+1558114us) service_pool.cc:225] Handling call
0831 13:58:56.656271 (+ 269us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.441477 (+785206us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.953725 (+512248us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b54003d54d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:57.953901 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1797617536}
I20250831 13:58:57.955305 23067 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:57.957674 22927 consensus_queue.cc:1046] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:57.951561 23058 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:57.954811 20496 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:57.918433 22900 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:57.869026 22901 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:57.960495 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.933873 20542 service_pool.cc:138] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:41400 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1399)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34436 (request call id 67)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1392)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1505)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1503)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1408)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1401)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1501)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1393)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34436 (request call id 66)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1407)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1391)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1403)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1406)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1465)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1466)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1467)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1468)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1469)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1470)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1472)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1474)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1476)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1477)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1479)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1481)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1482)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1483)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1492)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1493)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1388)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1389)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1495)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1499)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1502)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1395)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1396)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1402)
I20250831 13:58:57.962375 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:57.962425 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:57.967511 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1464) took 2104 ms. Trace:
I20250831 13:58:57.968667 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:57.969967 20500 raft_consensus.cc:1238] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
W20250831 13:58:57.980155 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 59100us
I20250831 13:58:57.971951 23000 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20250831 13:58:57.972851 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.976578 23045 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:57.961170 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.980947 20648 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:57.979806 20568 rpcz_store.cc:276] 0831 13:58:55.863156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.626913 (+1763757us) service_pool.cc:225] Handling call
0831 13:58:57.636730 (+ 9817us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:57.779483 (+142753us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:57.967301 (+187818us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54003b5954. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:57.967459 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":310945024}
I20250831 13:58:57.981333 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
W20250831 13:58:57.989769 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:57.991946 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:57.995007 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 185282us
I20250831 13:58:57.996834 20498 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:57.998785 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:57.999316 20497 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:58:58.000233 20501 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:57.991954 22780 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:58.000736 20498 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:58.001410 20501 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:58.002339 20498 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:57.999349 20568 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:58.003806 20498 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:58.002514 20501 raft_consensus.cc:1238] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:57.982844 23000 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:58.008564 23000 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:58.009639 22882 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.011871 22882 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:58.012912 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.013556 20573 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:58.016057 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:58.019093 20569 raft_consensus.cc:3058] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:58.020030 20541 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:58.018802 23083 consensus_queue.cc:1046] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20250831 13:58:58.026610 23083 raft_consensus.cc:3053] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:58.026912 22901 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:58.026988 23083 raft_consensus.cc:738] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.027274 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1463) took 2173 ms. Trace:
W20250831 13:58:58.028044 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 66888.1us
I20250831 13:58:58.027704 20571 rpcz_store.cc:276] 0831 13:58:55.853604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.018969 (+2165365us) service_pool.cc:225] Handling call
0831 13:58:58.026702 (+ 7733us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.026985 (+ 283us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:58.027007 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:58.027057 (+ 50us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:58.027098 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.027146 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:58.027228 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.030982 20496 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:58.040890 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:58.041468 22979 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:58.037483 22999 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:58.044694 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.045578 20574 raft_consensus.cc:2408] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20250831 13:58:58.046401 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.048185 20574 raft_consensus.cc:2391] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 5: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:58.048946 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:58.049039 20651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 1007) took 3761 ms (client timeout 2067 ms). Trace:
I20250831 13:58:58.049731 20574 raft_consensus.cc:2391] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:58.050575 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:58.049516 20651 rpcz_store.cc:269] 0831 13:58:54.287952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:54.396081 (+108129us) service_pool.cc:225] Handling call
0831 13:58:58.048827 (+3652746us) spinlock_profiling.cc:243] Waited 3.32 s on lock 0x7b54001b12d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:58.048971 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7642789760}
I20250831 13:58:58.051862 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.052580 20570 raft_consensus.cc:2391] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:58.053339 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.053815 20570 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:58.053568 23000 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:58.056159 23000 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:58:58.062045 22901 consensus_queue.cc:1046] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:58:58.062650 22660 consensus_queue.cc:1046] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:58.063562 20617 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:58.063668 22850 consensus_queue.cc:1046] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:58:58.063846 22660 consensus_queue.cc:1046] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:58.064453 23061 raft_consensus.cc:3053] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:58.028088 23083 consensus_queue.cc:260] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.064813 22850 consensus_queue.cc:1046] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:58.065490 22369 raft_consensus.cc:3053] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:58.065280 23013 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:58.061053 21176 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:58.066210 23013 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:58.060056 22853 raft_consensus.cc:3053] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:58.066017 22369 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.067358 23013 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:58.059489 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.058566 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.065393 23083 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:58.069507 20577 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:58.050616 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.064846 23061 raft_consensus.cc:738] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.070564 20574 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:58.066265 22850 raft_consensus.cc:3053] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:58.069309 20575 raft_consensus.cc:2408] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:58:58.068053 22369 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.065085 22822 raft_consensus.cc:3053] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:58.072293 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:58.067399 22853 raft_consensus.cc:738] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:58.075529 20573 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:58.063266 22901 consensus_queue.cc:1046] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20250831 13:58:58.076683 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.080679 20571 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:58:58.066874 22873 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:58.066509 22660 consensus_queue.cc:1046] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20250831 13:58:58.113988 20568 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:58:58.117076 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:58.117274 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:58.129968 20500 raft_consensus.cc:2391] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:58.130046 20495 raft_consensus.cc:2391] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:58.117761 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.118324 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.130779 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:58.118816 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.075095 22822 raft_consensus.cc:738] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.119896 22369 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:58.123018 22786 consensus_queue.cc:1046] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20250831 13:58:58.124168 23000 consensus_queue.cc:1046] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:58.124771 22371 consensus_queue.cc:1046] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:58:58.073817 22850 raft_consensus.cc:738] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.155655 20575 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:58:58.156100 22780 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250831 13:58:58.126667 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.065644 22767 raft_consensus.cc:3053] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:58.157807 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:58.132335 20496 raft_consensus.cc:2391] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 8: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:58:58.133177 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:58.159703 20499 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 10: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:58:58.133884 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 137108us
I20250831 13:58:58.139377 20494 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:58.160631 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.137419 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:58.139978 20572 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:58.151912 20573 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:58.125489 23003 consensus_queue.cc:1046] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:58:58.151752 22644 consensus_queue.cc:1046] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20250831 13:58:58.153038 22786 raft_consensus.cc:3053] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:58.152943 22822 consensus_queue.cc:260] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:58.155246 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.142027 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.162106 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.115053 22873 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:58.165553 20568 raft_consensus.cc:2391] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:58.166612 20497 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:58:58.184516 20571 raft_consensus.cc:2391] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 11: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:58.179847 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.161406 22767 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.197585 22371 raft_consensus.cc:3053] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:58.197782 22822 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:58.198073 22371 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.198513 22767 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.199188 22371 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.199918 22767 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:58.200291 22371 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:58.205509 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.211905 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.212710 20576 raft_consensus.cc:2391] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 10: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:58.210491 23000 raft_consensus.cc:3053] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:58.126077 22853 consensus_queue.cc:260] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.222723 23000 raft_consensus.cc:738] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.223528 22853 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:58.223707 23000 consensus_queue.cc:260] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.224865 23000 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:58.180114 23061 consensus_queue.cc:260] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.227731 23061 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:58.168938 22786 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:58.186865 22873 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:58.192021 22799 raft_consensus.cc:3053] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:58.161593 22999 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:58.248804 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 142001us
W20250831 13:58:58.260725 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 66149.9us
I20250831 13:58:58.266983 20543 leader_election.cc:400] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:58:58.256373 20397 catalog_manager.cc:5582] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 5 to 6, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 6 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:58.156347 22850 consensus_queue.cc:260] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:58.271006 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:58.276562 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:58.276933 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.275311 20575 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:58.288774 20617 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:58.288985 20576 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 8. Ops: []
I20250831 13:58:58.298357 20568 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:58.275523 20397 catalog_manager.cc:5582] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:58.257266 22799 raft_consensus.cc:738] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.303191 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1467) took 2426 ms. Trace:
I20250831 13:58:58.304927 20541 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:58:58.306783 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1466) took 2435 ms. Trace:
I20250831 13:58:58.343948 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.352016 20568 rpcz_store.cc:276] 0831 13:58:55.863816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.190533 (+2326717us) service_pool.cc:225] Handling call
0831 13:58:58.298101 (+107568us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.298916 (+ 815us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.299120 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:58.353734 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 99929.1us
W20250831 13:58:58.354018 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.353076 20576 rpcz_store.cc:276] 0831 13:58:55.864078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.235103 (+2371025us) service_pool.cc:225] Handling call
0831 13:58:58.282649 (+ 47546us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.289818 (+ 7169us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.290031 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:58.360443 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 71760.8us
I20250831 13:58:58.364653 20571 raft_consensus.cc:1238] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
W20250831 13:58:58.365931 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 61115.9us
W20250831 13:58:58.362200 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.377589 20573 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 2.
I20250831 13:58:58.391719 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1465) took 2505 ms. Trace:
I20250831 13:58:58.392366 20571 rpcz_store.cc:276] 0831 13:58:55.863522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.195204 (+2331682us) service_pool.cc:225] Handling call
0831 13:58:58.285872 (+ 90668us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.368690 (+ 82818us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.368895 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.295223 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
W20250831 13:58:58.393333 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.301682 22850 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:58.296720 23045 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:58.390515 20397 catalog_manager.cc:5582] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 5, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 5 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:58.405972 20575 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:58.407202 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.301234 22979 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:58.283138 22780 consensus_queue.cc:1046] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20250831 13:58:58.372054 22959 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:58:58.381892 22815 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:58:58.367949 22874 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:58.436687 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 68675.1us
I20250831 13:58:58.430481 20397 catalog_manager.cc:5582] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:58.407944 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:58.435050 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:58.440963 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.307497 22999 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.456089 20543 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:58.466352 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.470269 20576 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:58:58.333920 22786 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.480274 20397 catalog_manager.cc:5582] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 2 to 5, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 5 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } }
I20250831 13:58:58.494465 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.495198 20573 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:58.506748 20397 catalog_manager.cc:5582] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:58.502151 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.513530 20568 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 7. Ops: []
I20250831 13:58:58.472118 23013 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.528528 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.528992 20571 raft_consensus.cc:2408] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:58:58.530331 22338 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:58.454190 22815 raft_consensus.cc:695] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:58.555157 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.518658 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1469) took 2628 ms. Trace:
I20250831 13:58:58.468906 23045 consensus_queue.cc:1035] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20250831 13:58:58.478209 22979 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.584589 20576 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:58:58.607944 20577 consensus_meta.cc:287] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.538s user 0.009s sys 0.001s
I20250831 13:58:58.618441 20568 rpcz_store.cc:276] 0831 13:58:55.890274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.426511 (+2536237us) service_pool.cc:225] Handling call
0831 13:58:58.493215 (+ 66704us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.518342 (+ 25127us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.518592 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:58.603681 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.620230 20577 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:58.641919 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 236077us
I20250831 13:58:58.641942 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.642769 20543 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:58.474567 22799 consensus_queue.cc:260] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:58.562500 23095 log.cc:927] Time spent T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.424s user 0.001s sys 0.001s
I20250831 13:58:58.664813 20568 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:58.682052 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:58.687580 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:58.688138 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:58.688477 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.556772 22804 raft_consensus.cc:3053] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:58:58.581862 20397 catalog_manager.cc:5582] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 5 to 6, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 6 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
W20250831 13:58:58.713068 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.556429 22338 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.516497 22874 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:58.733415 22874 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:58.733206 20397 catalog_manager.cc:5582] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 7, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 7 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
W20250831 13:58:58.736899 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.707131 22804 raft_consensus.cc:738] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.737637 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 964) took 1305 ms. Trace:
I20250831 13:58:58.738003 20493 rpcz_store.cc:276] 0831 13:58:57.431914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.982982 (+551068us) service_pool.cc:225] Handling call
0831 13:58:57.995375 (+ 12393us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:58.052177 (+ 56802us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:58.052214 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:58.052837 (+ 623us) log.cc:844] Serialized 52 byte log entry
0831 13:58:58.160292 (+107455us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:58.215647 (+ 55355us) log.cc:844] Serialized 14 byte log entry
0831 13:58:58.241656 (+ 26009us) log.cc:844] Serialized 14 byte log entry
0831 13:58:58.242144 (+ 488us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.242243 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:58.737405 (+495162us) raft_consensus.cc:1683] finished
0831 13:58:58.737434 (+ 29us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:58.737566 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12426,"prepare.queue_time_us":55764,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":99,"raft.queue_time_us":15726,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":26390,"spinlock_wait_cycles":15236224,"thread_start_us":106972,"threads_started":1,"wal-append.queue_time_us":682124}
W20250831 13:58:58.742692 20569 consensus_meta.cc:287] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.669s user 0.008s sys 0.002s
I20250831 13:58:58.749336 20569 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
W20250831 13:58:58.749977 20569 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1487) took 1827 ms (client timeout 2214 ms). Trace:
W20250831 13:58:58.750377 20569 rpcz_store.cc:269] 0831 13:58:56.922239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:57.968370 (+1046131us) service_pool.cc:225] Handling call
0831 13:58:58.749910 (+781540us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.751372 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:58.742513 20644 consensus_meta.cc:287] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.666s user 0.006s sys 0.002s
I20250831 13:58:58.752135 20644 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:58.762495 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.762607 20466 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:58.763612 20569 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:58.765708 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.768045 20573 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:58:58.731102 23067 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:58:58.776448 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.739378 22882 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:58.738936 23013 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:58.746062 21208 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:58.786348 20397 catalog_manager.cc:5582] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 3, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:58.807616 22613 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
W20250831 13:58:58.807973 23100 log.cc:927] Time spent T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.446s user 0.002s sys 0.000s
I20250831 13:58:58.808247 22613 raft_consensus.cc:695] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:58.809369 22613 consensus_queue.cc:237] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:58.814477 20570 consensus_meta.cc:287] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.760s user 0.001s sys 0.009s
I20250831 13:58:58.815804 22338 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:58.819226 20570 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:58.819804 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1391) took 1054 ms. Trace:
I20250831 13:58:58.820151 20570 rpcz_store.cc:276] 0831 13:58:57.765323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.053115 (+287792us) service_pool.cc:225] Handling call
0831 13:58:58.819752 (+766637us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.738312 22804 consensus_queue.cc:260] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.719331 22815 consensus_queue.cc:237] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.827098 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.799168 22882 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.824224 20397 catalog_manager.cc:5582] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 4 to 8, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 8 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
W20250831 13:58:58.849864 20572 consensus_meta.cc:287] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.671s user 0.009s sys 0.001s
I20250831 13:58:58.850365 20572 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:58:58.851097 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1399) took 1072 ms. Trace:
I20250831 13:58:58.851207 20568 raft_consensus.cc:1238] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:58.792927 23067 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:58.852494 20572 rpcz_store.cc:276] 0831 13:58:57.779069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.075523 (+296454us) service_pool.cc:225] Handling call
0831 13:58:58.851044 (+775521us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:58.853837 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.820578 22999 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:58.855962 20577 raft_consensus.cc:1273] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:58:58.859402 20570 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:58:58.865173 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:58.867422 20570 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
W20250831 13:58:58.867728 20574 consensus_meta.cc:287] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.797s user 0.006s sys 0.005s
I20250831 13:58:58.868129 20574 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:58:58.871491 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1477) took 2841 ms. Trace:
I20250831 13:58:58.871932 20570 rpcz_store.cc:276] 0831 13:58:56.026408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.860056 (+2833648us) service_pool.cc:225] Handling call
0831 13:58:58.867218 (+ 7162us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.867897 (+ 679us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.868104 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.873378 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.874482 20574 raft_consensus.cc:2408] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:58:58.864741 22999 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:58.866783 22979 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:58.867292 20397 catalog_manager.cc:5582] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 4, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 4 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:58.888051 20467 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:58.891292 20572 raft_consensus.cc:2374] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7. Re-sending same reply.
I20250831 13:58:58.883867 20570 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
W20250831 13:58:58.883190 22853 consensus_meta.cc:287] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.659s user 0.010s sys 0.001s
I20250831 13:58:58.892273 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1479) took 2846 ms. Trace:
I20250831 13:58:58.893575 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1505) took 1061 ms. Trace:
I20250831 13:58:58.900647 20570 rpcz_store.cc:276] 0831 13:58:56.046216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.872845 (+2826629us) service_pool.cc:225] Handling call
0831 13:58:58.883641 (+ 10796us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.891950 (+ 8309us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.892184 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.895920 20571 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:58.904983 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:58.906010 20574 raft_consensus.cc:2391] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:58.891875 20397 catalog_manager.cc:5582] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
W20250831 13:58:58.895859 20501 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.500s user 0.000s sys 0.037s
I20250831 13:58:58.908988 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1506) took 1076 ms. Trace:
I20250831 13:58:58.913622 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1476) took 2887 ms. Trace:
I20250831 13:58:58.914067 20571 rpcz_store.cc:276] 0831 13:58:56.021808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.861544 (+2839736us) service_pool.cc:225] Handling call
0831 13:58:58.882398 (+ 20854us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.908926 (+ 26528us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.909166 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.900820 20498 rpcz_store.cc:276] 0831 13:58:57.832198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.032193 (+199995us) service_pool.cc:225] Handling call
0831 13:58:58.077485 (+ 45292us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:58.179019 (+101534us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:58.179062 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:58.216092 (+ 37030us) log.cc:844] Serialized 52 byte log entry
0831 13:58:58.707896 (+491804us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:58.707945 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.708008 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:58.893346 (+185338us) raft_consensus.cc:1683] finished
0831 13:58:58.893365 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:58.893510 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":103569,"raft.queue_time_us":102833,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":10602,"spinlock_wait_cycles":1871744,"thread_start_us":460271,"threads_started":1,"wal-append.queue_time_us":665650}
I20250831 13:58:58.907855 23035 raft_consensus.cc:3053] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:58.897017 22999 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:58.921785 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1471) took 3002 ms. Trace:
I20250831 13:58:58.922187 20568 rpcz_store.cc:276] 0831 13:58:55.919107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.675282 (+2756175us) service_pool.cc:225] Handling call
0831 13:58:58.851040 (+175758us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.851602 (+ 562us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.921594 (+ 69992us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b540031d8d0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:58.921730 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":272594816}
I20250831 13:58:58.928324 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1481) took 2866 ms. Trace:
I20250831 13:58:58.928755 20570 rpcz_store.cc:276] 0831 13:58:56.061661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.901784 (+2840123us) service_pool.cc:225] Handling call
0831 13:58:58.912292 (+ 10508us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.920583 (+ 8291us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:58.920619 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:58.928039 (+ 7420us) raft_consensus.cc:1643] Marking committed up to 7
0831 13:58:58.928088 (+ 49us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.928151 (+ 63us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:58.928274 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.909451 20501 rpcz_store.cc:276] 0831 13:58:57.832536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.019449 (+186913us) service_pool.cc:225] Handling call
0831 13:58:58.037782 (+ 18333us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:58.052054 (+ 14272us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:58.052083 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:58.052712 (+ 629us) log.cc:844] Serialized 52 byte log entry
0831 13:58:58.342121 (+289409us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:58.908457 (+566336us) log.cc:844] Serialized 14 byte log entry
0831 13:58:58.908686 (+ 229us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.908754 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:58:58.908801 (+ 47us) raft_consensus.cc:1683] finished
0831 13:58:58.908811 (+ 10us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:58.908934 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13908,"prepare.queue_time_us":562724,"raft.queue_time_us":14293,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"thread_start_us":801409,"threads_started":2,"wal-append.queue_time_us":764872}
I20250831 13:58:58.915336 22853 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 13:58:58.933306 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:58.922416 22369 consensus_meta.cc:287] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.762s user 0.008s sys 0.004s
I20250831 13:58:58.918442 23035 raft_consensus.cc:738] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:58.945072 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1489) took 2019 ms. Trace:
W20250831 13:58:58.919274 21174 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.018s sys 0.056s
I20250831 13:58:58.955420 20570 rpcz_store.cc:276] 0831 13:58:56.925685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.929918 (+2004233us) service_pool.cc:225] Handling call
0831 13:58:58.944567 (+ 14649us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.944774 (+ 207us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:58.944794 (+ 20us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:58.944846 (+ 52us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:58.944871 (+ 25us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.944935 (+ 64us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:58.945025 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:58.945989 20568 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
W20250831 13:58:58.949312 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:58.950964 22229 raft_consensus.cc:3053] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Stepping down as leader of term 3
I20250831 13:58:58.963012 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1484) took 2052 ms. Trace:
I20250831 13:58:58.963495 20568 rpcz_store.cc:276] 0831 13:58:56.910429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.923414 (+2012985us) service_pool.cc:225] Handling call
0831 13:58:58.945804 (+ 22390us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.956958 (+ 11154us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.962946 (+ 5988us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:58.967427 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 79486.4us
I20250831 13:58:58.920967 20397 catalog_manager.cc:5582] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 6, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 6 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
W20250831 13:58:58.973765 23058 consensus_meta.cc:287] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 1.015s user 0.008s sys 0.004s
I20250831 13:58:58.933151 22882 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:58.975212 20571 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
W20250831 13:58:58.958323 21174 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.549s user 0.019s sys 0.057s
I20250831 13:58:58.993650 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:58.995514 20467 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:58.975950 20576 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:58:58.936467 22853 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:58.977948 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1483) took 2068 ms. Trace:
I20250831 13:58:58.980275 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1470) took 3080 ms. Trace:
I20250831 13:58:58.996876 20577 rpcz_store.cc:276] 0831 13:58:55.899630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.663458 (+2763828us) service_pool.cc:225] Handling call
0831 13:58:58.845808 (+182350us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.857843 (+ 12035us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.980092 (+122249us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54000e7750. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:58.980218 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316351360}
I20250831 13:58:58.994239 20499 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:58.998176 20571 rpcz_store.cc:276] 0831 13:58:56.910069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.918563 (+2008494us) service_pool.cc:225] Handling call
0831 13:58:58.966307 (+ 47744us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.977657 (+ 11350us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:58.977886 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:58.982347 22850 consensus_meta.cc:287] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.568s user 0.006s sys 0.000s
W20250831 13:58:58.975659 23061 consensus_meta.cc:287] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.747s user 0.006s sys 0.006s
I20250831 13:58:58.988876 20493 raft_consensus.cc:1238] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:59.009936 20568 raft_consensus.cc:1238] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 1. Current term is 2. Ops: []
I20250831 13:58:58.990444 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:59.011103 20501 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 8
I20250831 13:58:58.992056 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:58:58.993096 22873 consensus_meta.cc:287] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.700s user 0.005s sys 0.002s
I20250831 13:58:58.999568 20574 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:59.007820 23067 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:58:59.008195 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.015518 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1490) took 2086 ms. Trace:
I20250831 13:58:59.016376 20397 catalog_manager.cc:5582] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 5, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 5 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.011857 20497 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:58.962713 22229 raft_consensus.cc:738] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:59.014227 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1492) took 2057 ms. Trace:
I20250831 13:58:59.021941 20568 rpcz_store.cc:276] 0831 13:58:56.956478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.964321 (+2007843us) service_pool.cc:225] Handling call
0831 13:58:59.009724 (+ 45403us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.010563 (+ 839us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.014156 (+ 3593us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:59.008697 22371 consensus_meta.cc:287] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.808s user 0.006s sys 0.006s
I20250831 13:58:59.015990 20570 rpcz_store.cc:276] 0831 13:58:56.929447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.956567 (+2027120us) service_pool.cc:225] Handling call
0831 13:58:58.994375 (+ 37808us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.015127 (+ 20752us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.015165 (+ 38us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.015215 (+ 50us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:59.015260 (+ 45us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.015330 (+ 70us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.015458 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.018374 20397 catalog_manager.cc:5582] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 5, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 5 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.024854 20496 raft_consensus.cc:1238] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 3. Current term is 4. Ops: []
I20250831 13:58:59.028249 20570 raft_consensus.cc:1273] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:59.030026 20498 raft_consensus.cc:1273] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:58:59.031277 20496 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20250831 13:58:59.032756 20576 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:59.033970 20617 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:59.034811 20572 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 10. Current term is 11. Ops: []
I20250831 13:58:59.040825 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1495) took 1500 ms. Trace:
I20250831 13:58:59.006386 22853 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:59.041217 20572 rpcz_store.cc:276] 0831 13:58:57.540726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.018744 (+1478018us) service_pool.cc:225] Handling call
0831 13:58:59.034587 (+ 15843us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.040546 (+ 5959us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.040764 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.036967 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1496) took 1495 ms. Trace:
I20250831 13:58:59.037356 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1499) took 1486 ms. Trace:
I20250831 13:58:59.043082 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1482) took 2974 ms. Trace:
I20250831 13:58:59.039378 20495 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:59.043836 20574 rpcz_store.cc:276] 0831 13:58:56.068286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.906704 (+2838418us) service_pool.cc:225] Handling call
0831 13:58:58.999338 (+ 92634us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.016013 (+ 16675us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.042871 (+ 26858us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b54003b4550. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:59.043020 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68127104}
W20250831 13:58:59.045570 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 77294.6us
I20250831 13:58:59.046339 20467 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.038338 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1493) took 2074 ms. Trace:
W20250831 13:58:58.995218 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 613885us
I20250831 13:58:59.043319 20570 rpcz_store.cc:276] 0831 13:58:57.542222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.027521 (+1485299us) service_pool.cc:225] Handling call
0831 13:58:59.027840 (+ 319us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.028707 (+ 867us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.028910 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.042593 20568 rpcz_store.cc:276] 0831 13:58:57.541126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.023009 (+1481883us) service_pool.cc:225] Handling call
0831 13:58:59.023374 (+ 365us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.036587 (+ 13213us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.036623 (+ 36us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.036675 (+ 52us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:59.036716 (+ 41us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.036784 (+ 68us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.036897 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.009446 22979 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:59.055943 20570 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:59.050300 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1388) took 1702 ms. Trace:
I20250831 13:58:59.056737 20577 rpcz_store.cc:276] 0831 13:58:57.348225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.009629 (+1661404us) service_pool.cc:225] Handling call
0831 13:58:59.009985 (+ 356us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.049988 (+ 40003us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.050017 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.050054 (+ 37us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:58:59.050085 (+ 31us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.050150 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.050254 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.030663 22999 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:58:59.047269 20576 rpcz_store.cc:276] 0831 13:58:56.964408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.014629 (+2050221us) service_pool.cc:225] Handling call
0831 13:58:59.032552 (+ 17923us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.038050 (+ 5498us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.038263 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.065385 20568 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:59.037533 20397 catalog_manager.cc:5582] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 5 to 7, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 7 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.030838 22882 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.050087 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1389) took 1631 ms. Trace:
W20250831 13:58:59.068079 20541 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s user 0.015s sys 0.060s
I20250831 13:58:59.050200 20499 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:59.049504 22371 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:59.052474 23035 consensus_queue.cc:260] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.023835 22873 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:59.059721 22229 consensus_queue.cc:260] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.067811 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1504) took 1367 ms. Trace:
I20250831 13:58:59.074935 20568 rpcz_store.cc:276] 0831 13:58:57.702187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.056354 (+1354167us) service_pool.cc:225] Handling call
0831 13:58:59.065136 (+ 8782us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.067601 (+ 2465us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.067758 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.068259 20571 rpcz_store.cc:276] 0831 13:58:57.418983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.011241 (+1592258us) service_pool.cc:225] Handling call
0831 13:58:59.042156 (+ 30915us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.049730 (+ 7574us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.049754 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.049797 (+ 43us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:59.049837 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.049911 (+ 74us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.050027 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.077167 20397 catalog_manager.cc:5582] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 6 to 7, leader changed from ab7ade51fa134a1b814fab748f01d43d (127.18.233.132) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 7 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:58:59.077914 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1400) took 1298 ms. Trace:
I20250831 13:58:59.084148 20571 rpcz_store.cc:276] 0831 13:58:57.779352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.076850 (+1297498us) service_pool.cc:225] Handling call
0831 13:58:59.077245 (+ 395us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.077536 (+ 291us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.077570 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.077640 (+ 70us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:59.077682 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.077747 (+ 65us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.077855 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.084910 20397 catalog_manager.cc:5582] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.092767 20501 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:59.071134 22371 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:59.096161 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 453475us
W20250831 13:58:59.068907 20541 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.702s user 0.016s sys 0.061s
I20250831 13:58:59.068655 20617 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:59.097083 20543 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.058440 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1502) took 1434 ms. Trace:
I20250831 13:58:59.098359 20570 rpcz_store.cc:276] 0831 13:58:57.626379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.055436 (+1429057us) service_pool.cc:225] Handling call
0831 13:58:59.055740 (+ 304us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.058157 (+ 2417us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.058384 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:58:59.094367 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:58:59.083714 22780 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.036s sys 0.063s
W20250831 13:58:59.102892 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 736094us
I20250831 13:58:59.101480 20651 raft_consensus.cc:2391] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 10: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:59.102197 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.066835 22999 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.105887 22999 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.106421 22999 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.106702 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.106920 22999 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.107800 20467 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.073268 22882 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.106715 23075 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:59.109814 23075 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:59.109675 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.110590 23075 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:59.110666 20651 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 3.
I20250831 13:58:59.108927 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.094923 22229 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:59.113250 20467 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.071274 22979 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.072957 23067 consensus_queue.cc:1046] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20250831 13:58:59.096856 23035 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:59.111769 22554 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:59.117367 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1402) took 1337 ms. Trace:
I20250831 13:58:59.118407 20543 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.097784 21208 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:58:59.136026 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:59.103164 22338 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:59.116456 22859 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:58:59.138161 22338 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:59.119901 23083 consensus_meta.cc:287] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 1.050s user 0.005s sys 0.009s
I20250831 13:58:59.119827 20467 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.136597 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1409) took 1140 ms. Trace:
I20250831 13:58:59.139083 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.134536 22927 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:58:59.149000 20467 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:59.133702 22767 consensus_meta.cc:287] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.933s user 0.014s sys 0.000s
I20250831 13:58:59.133947 20571 rpcz_store.cc:276] 0831 13:58:57.779917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.085625 (+1305708us) service_pool.cc:225] Handling call
0831 13:58:59.102477 (+ 16852us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.113647 (+ 11170us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.113684 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.116146 (+ 2462us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:59.117035 (+ 889us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.117134 (+ 99us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.117305 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:59.150499 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.115453 20397 catalog_manager.cc:5582] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 2 to 4, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 4 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.116922 23075 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:58:59.098575 22371 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:58:59.139896 23000 consensus_meta.cc:287] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.915s user 0.006s sys 0.007s
I20250831 13:58:59.108778 22819 consensus_queue.cc:1046] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20250831 13:58:59.153772 22859 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:58:59.147275 20570 rpcz_store.cc:276] 0831 13:58:57.995741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.102699 (+1106958us) service_pool.cc:225] Handling call
0831 13:58:59.103047 (+ 348us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.133438 (+ 30391us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.133477 (+ 39us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.136340 (+ 2863us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:58:59.136394 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.136448 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.136551 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:58:59.103909 22780 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.036s sys 0.063s
I20250831 13:58:59.108786 22999 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.162585 23075 raft_consensus.cc:3053] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:58:59.162796 22999 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:58:59.163031 23075 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:59.163786 22999 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:59.163925 23075 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.167179 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.167934 20648 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:59.168475 20571 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:59.170861 20543 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.172120 22859 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250831 13:58:59.137050 22882 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.154093 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.172690 20397 catalog_manager.cc:5582] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 2 to 4, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 4 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.179986 20644 raft_consensus.cc:2391] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:58:59.147987 20617 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:59.146704 22338 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:59.181694 20651 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:59.154804 22927 raft_consensus.cc:695] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:59.150884 22900 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:58:59.145339 22822 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.012s
I20250831 13:58:59.161625 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:58:59.162050 21175 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.047s sys 0.059s
I20250831 13:58:59.172083 20543 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.151719 20467 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:58:59.188727 21175 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.048s sys 0.060s
I20250831 13:58:59.172716 22780 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:59.157893 20645 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:58:59.180521 20397 catalog_manager.cc:5582] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 5 to 7, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 7 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
W20250831 13:58:59.181905 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.194550 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.195282 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.146373 23013 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20250831 13:58:59.196476 20543 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:58:59.190181 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.199105 20502 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 10. Current term is 11. Ops: []
I20250831 13:58:59.200058 20499 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 4. Current term is 5. Ops: []
I20250831 13:58:59.199599 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.193457 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.188510 20646 raft_consensus.cc:2408] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20250831 13:58:59.206231 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.208454 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.210389 20467 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.202653 22859 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:58:59.212752 22819 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250831 13:58:59.212399 22927 consensus_queue.cc:237] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.213302 22987 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:59.213747 22987 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:59.214179 22987 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:59.186983 22900 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:59.215015 22900 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:59.186177 22369 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:58:59.228428 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:59.231109 20467 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.201681 23067 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.202165 23045 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:58:59.234041 20397 catalog_manager.cc:5582] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 6, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 6 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:59.225831 22873 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:59.241015 20397 catalog_manager.cc:5582] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
W20250831 13:58:59.244102 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 47720.1us
I20250831 13:58:59.245324 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.233728 22859 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:58:59.246361 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.251320 20543 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.223533 23058 consensus_queue.cc:1046] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.020s
I20250831 13:58:59.235451 22369 raft_consensus.cc:695] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:59.254189 20467 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.243129 22979 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.260418 20397 catalog_manager.cc:5582] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 2 to 5, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 5 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.222563 22987 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.259217 22859 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:58:59.265208 22819 raft_consensus.cc:2802] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Leader election won for term 2
I20250831 13:58:59.256541 22767 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.269178 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1395) took 1502 ms. Trace:
I20250831 13:58:59.269655 20467 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.242132 23067 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.259232 22780 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:58:59.272996 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.273948 20576 rpcz_store.cc:276] 0831 13:58:57.772021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.064617 (+1292596us) service_pool.cc:225] Handling call
0831 13:58:59.076487 (+ 11870us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.110915 (+ 34428us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.110944 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.110987 (+ 43us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:59.111014 (+ 27us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.268955 (+157941us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b54002fe754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:58:59.269005 (+ 50us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.269116 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78728064}
I20250831 13:58:59.297866 20543 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.303342 20467 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.304721 20467 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.306744 22979 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:58:59.301836 20466 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.504s user 0.009s sys 0.013s
W20250831 13:58:59.310346 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 512856us
I20250831 13:58:59.307511 20397 catalog_manager.cc:5582] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 6 to 8, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 8 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250831 13:58:59.301223 22799 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:59.309633 22859 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
W20250831 13:58:59.330791 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.350765 22859 raft_consensus.cc:695] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:58:59.351397 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.351634 22859 consensus_queue.cc:237] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.313557 22369 consensus_queue.cc:237] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.336093 23067 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:59.337747 22900 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.356549 22900 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.357225 22900 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:58:59.338613 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.358320 22900 raft_consensus.cc:695] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:59.359319 22900 consensus_queue.cc:237] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.347777 22987 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.350225 22554 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:59.366227 22554 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:58:59.366612 22554 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:58:59.368541 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.369211 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.371819 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:59.372411 20572 raft_consensus.cc:2374] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5. Re-sending same reply.
I20250831 13:58:59.338070 22338 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.374814 22338 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:59.363026 22786 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:59.377503 21208 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47742 (request call id 106) took 1197 ms. Trace:
I20250831 13:58:59.377892 21208 rpcz_store.cc:276] 0831 13:58:58.180459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.204935 (+ 24476us) service_pool.cc:225] Handling call
0831 13:58:59.377297 (+1172362us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b54005e1ad4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:59.377425 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":747077888}
I20250831 13:58:59.378301 22780 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.379006 22780 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.379842 22959 raft_consensus.cc:3053] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:58:59.380218 22959 raft_consensus.cc:738] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:59.380971 22959 consensus_queue.cc:260] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.381290 22780 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.381544 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:59.382108 22959 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:59.384147 20574 raft_consensus.cc:1238] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:58:59.384904 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1500) took 1842 ms. Trace:
I20250831 13:58:59.385257 20574 rpcz_store.cc:276] 0831 13:58:57.542620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.044901 (+1502281us) service_pool.cc:225] Handling call
0831 13:58:59.383736 (+338835us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b58000f0398. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d75ce 000000000035a3c9 00007fb9c185e300 00007fb9c1924ec4 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:59.383929 (+ 193us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.384615 (+ 686us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.384842 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5375232}
I20250831 13:58:59.288179 22767 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.391650 23107 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.392195 23107 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:59.392699 22767 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:59.393546 20648 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:59.394765 22338 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.395224 22338 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.398999 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:59.382184 20501 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:59.364410 23045 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250831 13:58:59.364997 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:59.400719 23003 raft_consensus.cc:3053] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 1 LEADER]: Stepping down as leader of term 1
I20250831 13:58:59.400787 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.402045 20497 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:59.400835 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:59.408655 20617 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:59.407629 20496 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:58:59.414103 20617 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:59.402189 20647 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:58:59.407228 20397 catalog_manager.cc:5582] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 5, leader changed from 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 5 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
W20250831 13:58:59.417315 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.418668 20574 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:59.418835 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:59.420701 20572 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:59.420296 20397 catalog_manager.cc:5582] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.419531 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:59.424669 20576 raft_consensus.cc:2391] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:59.425544 20575 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.747s user 0.019s sys 0.068s
I20250831 13:58:59.426432 20617 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:59.427747 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 103388us
W20250831 13:58:59.425876 20575 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.761s user 0.019s sys 0.068s
I20250831 13:58:59.428267 20397 catalog_manager.cc:5582] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 3 to 4, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 4 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.432570 20543 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.407122 23003 raft_consensus.cc:738] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:59.416625 23058 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:58:59.437288 20397 catalog_manager.cc:5582] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 1 to 3, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 212b66976f874aa682cae0fb77928786 (127.18.233.130). New cstate: current_term: 3 leader_uuid: "212b66976f874aa682cae0fb77928786" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:58:59.425866 23013 raft_consensus.cc:2802] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader election won for term 10
I20250831 13:58:59.429762 23000 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:59.431372 22338 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:59.472265 20501 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:58:59.478637 20617 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:59.476433 20397 catalog_manager.cc:5582] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 reported cstate change: term changed from 2 to 3, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 3 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:58:59.458144 20494 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:59.481848 23075 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:59.436619 22804 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:58:59.484418 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.485037 20648 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 5.
W20250831 13:58:59.489724 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.491297 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:59.492354 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:59.493039 20572 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:59.496022 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:59.441329 23058 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:59.500612 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:58:59.500748 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 72214.8us
I20250831 13:58:59.476895 23000 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:59.521634 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.532357 20494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1514) took 1351 ms (client timeout 1781 ms). Trace:
W20250831 13:58:59.532794 20494 rpcz_store.cc:269] 0831 13:58:58.181170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.751122 (+569952us) service_pool.cc:225] Handling call
0831 13:58:59.532182 (+781060us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b540035b1d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:58:59.532297 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1554881920}
I20250831 13:58:59.478026 23013 raft_consensus.cc:695] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [term 10 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:59.535717 20570 raft_consensus.cc:2374] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 44d163094add49049cc749547d891602 in term 5. Re-sending same reply.
I20250831 13:58:59.539041 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:59.536060 20617 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:59.542557 20573 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.019s sys 0.028s
W20250831 13:58:59.543006 20573 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.681s user 0.020s sys 0.028s
I20250831 13:58:59.497525 22767 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:59.545790 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:59.546496 20570 raft_consensus.cc:3058] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:58:59.496876 22338 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.441850 23045 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:59.499475 23058 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:59.532366 23000 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:58:59.559794 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.568764 20495 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 9. Current term is 10. Ops: []
W20250831 13:58:59.575414 20616 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.019s sys 0.025s
W20250831 13:58:59.575738 20616 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.673s user 0.019s sys 0.025s
W20250831 13:58:59.575990 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 680211us
I20250831 13:58:59.578109 20543 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.565902 23067 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
W20250831 13:58:59.583132 22613 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.025s sys 0.033s
I20250831 13:58:59.563246 22767 raft_consensus.cc:695] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:58:59.597218 20569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.016s sys 0.012s
W20250831 13:58:59.597637 20569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.016s sys 0.012s
W20250831 13:58:59.611474 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.612028 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.613611 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:59.561733 22338 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:59.591060 22901 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.006s sys 0.021s
W20250831 13:58:59.546219 22815 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s user 0.013s sys 0.018s
W20250831 13:58:59.597116 22613 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.025s sys 0.033s
I20250831 13:58:59.429868 23107 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.628508 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.607178 22999 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:58:59.633585 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.634368 20648 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:58:59.550415 23003 consensus_queue.cc:260] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:59.639925 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 61921.8us
I20250831 13:58:59.615005 23067 raft_consensus.cc:695] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:59.640719 20572 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:59.640843 20543 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.639240 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.589769 22554 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:59.627530 22901 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.639s user 0.006s sys 0.021s
I20250831 13:58:59.663025 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:59.666916 20572 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:59.662287 20570 raft_consensus.cc:2466] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:58:59.634207 22882 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
W20250831 13:58:59.636181 22815 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.795s user 0.013s sys 0.019s
I20250831 13:58:59.679293 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:58:59.680320 20570 raft_consensus.cc:2391] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:58:59.665846 23035 consensus_meta.cc:287] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.533s user 0.008s sys 0.001s
I20250831 13:58:59.683195 23108 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:58:59.675158 22780 raft_consensus.cc:2747] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:58:59.684365 22338 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:59.687001 20651 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:58:59.694968 20541 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.695974 22780 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:58:59.701109 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.704063 22780 raft_consensus.cc:2747] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:58:59.705560 20541 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:59.707912 23108 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:59.708508 23108 raft_consensus.cc:695] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:59.709030 23058 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.709441 23108 consensus_queue.cc:237] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.711476 20541 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:59.712847 20571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.010s sys 0.022s
W20250831 13:58:59.713177 20571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.540s user 0.010s sys 0.022s
I20250831 13:58:59.669725 23045 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:58:59.713701 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:59.725442 20570 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 5.
I20250831 13:58:59.723519 23058 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:59.615615 23013 consensus_queue.cc:237] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:59.673122 23061 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.018s sys 0.015s
I20250831 13:58:59.738135 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:58:59.738621 20568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.003s sys 0.032s
W20250831 13:58:59.739172 20568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s user 0.003s sys 0.032s
I20250831 13:58:59.739301 20570 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:58:59.747555 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:58:59.748351 20570 raft_consensus.cc:2391] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 10: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:58:59.747902 20502 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.008s sys 0.015s
W20250831 13:58:59.750402 20502 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s user 0.008s sys 0.015s
I20250831 13:58:59.713747 23003 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 1 FOLLOWER]: Advancing to term 2
I20250831 13:58:59.763686 20572 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
W20250831 13:58:59.766628 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 125866us
W20250831 13:58:59.745899 22660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.017s sys 0.017s
I20250831 13:58:59.724100 22780 raft_consensus.cc:2762] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20250831 13:58:59.738750 23061 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.640s user 0.018s sys 0.015s
I20250831 13:58:59.725178 23045 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:58:59.764178 22927 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.020s sys 0.020s
I20250831 13:58:59.771546 20543 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:58:59.681666 22882 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:58:59.771481 22660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.557s user 0.017s sys 0.018s
W20250831 13:58:59.786759 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 482111us
I20250831 13:58:59.728435 22613 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.769743 20570 raft_consensus.cc:1273] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:58:59.768826 20572 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:58:59.738090 22979 consensus_queue.cc:1046] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20250831 13:58:59.811530 20467 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:58:59.786957 22927 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s user 0.020s sys 0.020s
I20250831 13:58:59.710897 22767 consensus_queue.cc:237] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.731758 23000 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:59.788371 22229 consensus_meta.cc:287] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.668s user 0.010s sys 0.000s
I20250831 13:58:59.782137 23045 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:58:59.824209 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.812913 22853 consensus_meta.cc:287] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.750s user 0.008s sys 0.003s
W20250831 13:58:59.846711 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 270935us
W20250831 13:58:59.849813 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.850252 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.850725 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.841795 22554 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.808804 22819 raft_consensus.cc:695] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:59.851589 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1472) took 3927 ms. Trace:
I20250831 13:58:59.857947 22819 consensus_queue.cc:237] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.858510 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:59.858462 20573 rpcz_store.cc:276] 0831 13:58:55.924101 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.770071 (+2845970us) service_pool.cc:225] Handling call
0831 13:58:58.801720 (+ 31649us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.802117 (+ 397us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:58:59.543424 (+741307us) log.cc:844] Serialized 14 byte log entry
0831 13:58:59.851148 (+307724us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.851215 (+ 67us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:59.851283 (+ 68us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.851379 (+ 96us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.851525 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":896858,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"thread_start_us":968769,"threads_started":2}
I20250831 13:58:59.814711 22999 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 13:58:59.852005 20499 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.018s sys 0.049s
W20250831 13:58:59.860849 20499 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s user 0.018s sys 0.064s
W20250831 13:58:59.826179 22371 consensus_meta.cc:287] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.667s user 0.012s sys 0.000s
I20250831 13:58:59.862151 22371 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:58:59.862609 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.864643 22371 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:58:59.854804 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:58:59.865571 20494 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:58:59.856009 22799 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.866029 20466 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:58:59.856091 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.860054 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:58:59.867628 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.868891 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:59.811746 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1468) took 3937 ms. Trace:
I20250831 13:58:59.869571 20494 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:58:59.872717 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 160133us
I20250831 13:58:59.873378 20541 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:58:59.874022 20496 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:58:59.874655 20541 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:58:59.875411 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.876214 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.876327 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 81926.7us
I20250831 13:58:59.877079 20467 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.879105 20617 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:59.879379 20495 raft_consensus.cc:1238] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 10. Current term is 11. Ops: []
I20250831 13:58:59.839428 22780 raft_consensus.cc:3053] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:58:59.866816 22987 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:58:59.885838 21175 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:58:59.884675 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:59.870165 20575 rpcz_store.cc:276] 0831 13:58:55.874345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.419247 (+2544902us) service_pool.cc:225] Handling call
0831 13:58:58.486094 (+ 66847us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:58.552358 (+ 66264us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 13:58:59.430273 (+877915us) log.cc:844] Serialized 14 byte log entry
0831 13:58:59.811402 (+381129us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:58:59.811455 (+ 53us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:58:59.811499 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.811570 (+ 71us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:58:59.811696 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1117323,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":353306,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":30329,"thread_start_us":1091798,"threads_started":2}
I20250831 13:58:59.899792 20496 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:58:59.890769 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.866564 22799 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.787384 23067 consensus_queue.cc:237] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.869454 22927 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:58:59.907068 20494 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:58:59.904021 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.815086 22613 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:59.875578 23022 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:58:59.910534 21175 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:58:59.882517 22999 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:58:59.918517 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:59.922407 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:59.925062 20467 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.897780 22987 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:59.877535 22229 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20250831 13:58:59.904713 22371 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:58:59.927510 23022 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.932911 22371 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:59.933030 22229 raft_consensus.cc:3053] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Stepping down as leader of term 9
I20250831 13:58:59.933379 22371 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:58:59.933434 22229 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:59.933682 22660 raft_consensus.cc:3053] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 10 LEADER]: Stepping down as leader of term 10
I20250831 13:58:59.934106 22660 raft_consensus.cc:738] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:58:59.934273 23022 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:58:59.931018 20643 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:58:59.887279 22780 raft_consensus.cc:738] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:58:59.927507 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:58:59.943254 20493 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 5.
I20250831 13:58:59.867173 22853 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:58:59.920908 22999 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:59.924086 22873 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:58:59.965281 20617 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:58:59.970923 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:58:59.971765 20648 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:58:59.972028 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:58:59.972599 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.943670 22987 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:58:59.924006 23127 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:58:59.981602 20500 raft_consensus.cc:1238] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:58:59.935171 23126 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:58:59.984685 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 216517us
I20250831 13:58:59.923596 22369 consensus_queue.cc:1046] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
W20250831 13:58:59.987545 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 71140.2us
I20250831 13:58:59.934378 22229 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.999262 20543 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:59:00.001130 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:58:59.967664 22873 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:58:59.967646 22999 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:58:59.983384 23083 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:58:59.981775 23107 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250831 13:59:00.006711 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 81735.2us
I20250831 13:59:00.014489 20467 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:58:59.977695 22927 raft_consensus.cc:695] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:58:59.983063 22786 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:00.029349 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1520) took 1046 ms. Trace:
I20250831 13:59:00.030284 20500 rpcz_store.cc:276] 0831 13:58:58.982748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.995435 (+ 12687us) service_pool.cc:225] Handling call
0831 13:58:58.995681 (+ 246us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.982028 (+986347us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:00.029080 (+ 47052us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b5400011dd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:00.029286 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2255952128}
I20250831 13:59:00.030815 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:58:59.998725 22901 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.031347 20498 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:59:00.005344 23112 log.cc:927] Time spent T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20250831 13:59:00.034276 20467 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:00.001688 23127 raft_consensus.cc:695] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:00.005568 22873 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:59:00.040936 23124 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20250831 13:59:00.035702 23109 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 13:59:00.041272 23000 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.041391 23111 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
W20250831 13:59:00.043310 23116 log.cc:927] Time spent T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.201s user 0.000s sys 0.001s
W20250831 13:59:00.044850 22959 consensus_meta.cc:287] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.662s user 0.003s sys 0.006s
W20250831 13:59:00.045545 23122 log.cc:927] Time spent T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
W20250831 13:59:00.046339 23130 log.cc:927] Time spent T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20250831 13:59:00.046433 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:00.085132 23141 raft_consensus.cc:2762] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20250831 13:59:00.085981 23141 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.086409 23141 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.987634 22660 consensus_queue.cc:260] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.088030 22660 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:59:00.088413 23141 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.090318 23141 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:00.046972 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1424) took 1289 ms. Trace:
I20250831 13:59:00.036929 22901 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:58:59.999679 23075 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20250831 13:59:00.028461 22882 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:00.052045 23126 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:59:00.054916 22338 consensus_queue.cc:1046] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 13:59:00.055634 22927 consensus_queue.cc:237] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.035030 22229 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:00.033183 22786 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:59:00.042881 23115 log.cc:927] Time spent T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20250831 13:59:00.041764 22850 raft_consensus.cc:3053] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
W20250831 13:59:00.097841 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 63632.4us
I20250831 13:59:00.094657 23141 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:00.102133 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1394) took 2335 ms. Trace:
W20250831 13:59:00.102916 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 192480us
I20250831 13:59:00.030061 23083 raft_consensus.cc:695] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:00.091225 20571 rpcz_store.cc:276] 0831 13:58:58.757821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.169780 (+411959us) service_pool.cc:225] Handling call
0831 13:58:59.170082 (+ 302us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:59.170276 (+ 194us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.170299 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:59.171142 (+ 843us) log.cc:844] Serialized 26 byte log entry
0831 13:58:59.713638 (+542496us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:59.748246 (+ 34608us) log.cc:844] Serialized 14 byte log entry
0831 13:58:59.748460 (+ 214us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.748528 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:00.046755 (+298227us) raft_consensus.cc:1683] finished
0831 13:59:00.046776 (+ 21us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:00.046902 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33463,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":5578,"raft.queue_time_us":34150,"raft.run_wall_time_us":24190,"spinlock_wait_cycles":8949504,"thread_start_us":540643,"threads_started":1,"wal-append.queue_time_us":812687}
I20250831 13:59:00.028932 22780 consensus_queue.cc:260] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.117045 20576 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:00.117204 20496 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:00.119189 23000 raft_consensus.cc:3053] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:00.119769 22853 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:00.121788 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1474) took 4155 ms. Trace:
W20250831 13:59:00.091138 23058 consensus_queue.cc:1162] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "58aded50ab0d4609a053647eb11b1f48" responder_term: 11 status { last_received { term: 10 index: 5 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 10. Current term is 11. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20250831 13:59:00.129132 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.097842 23111 raft_consensus.cc:695] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:00.132839 20467 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:00.123063 23127 consensus_queue.cc:237] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:00.137615 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.006s sys 0.013s
W20250831 13:59:00.164141 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.539s user 0.006s sys 0.013s
W20250831 13:59:00.164599 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:00.165032 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:00.165511 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.108898 22799 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.145442 20569 rpcz_store.cc:276] 0831 13:58:55.966414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:58.776543 (+2810129us) service_pool.cc:225] Handling call
0831 13:58:58.861326 (+ 84783us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:58:58.903357 (+ 42031us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:58.903397 (+ 40us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:58:58.918954 (+ 15557us) log.cc:844] Serialized 26 byte log entry
0831 13:58:59.598041 (+679087us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:58:59.598094 (+ 53us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.598202 (+ 108us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:00.121593 (+523391us) raft_consensus.cc:1683] finished
0831 13:59:00.121613 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:00.121734 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":19592,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":3019,"thread_start_us":628554,"threads_started":1,"wal-append.queue_time_us":838251}
I20250831 13:59:00.117954 22786 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:00.124722 22850 raft_consensus.cc:738] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:00.159792 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.163089 20497 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:00.168711 20573 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:59:00.167068 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 185257us
W20250831 13:59:00.165815 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:00.169395 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:00.169791 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.138818 23141 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:00.170624 20466 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.175346 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:00.175709 22900 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:00.176283 20647 raft_consensus.cc:2391] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:00.177587 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.178406 20572 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:59:00.179389 23142 log.cc:927] Time spent T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.067s user 0.003s sys 0.000s
I20250831 13:59:00.180138 22901 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:00.180897 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 56145.5us
I20250831 13:59:00.181564 20467 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:00.182557 22853 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:00.182669 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:00.183115 22853 raft_consensus.cc:695] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:00.184136 22853 consensus_queue.cc:237] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:00.185892 20616 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.186065 22901 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.186810 22799 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:00.187371 22799 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.187973 23126 raft_consensus.cc:2762] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20250831 13:59:00.189283 22799 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.189777 22799 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:00.190893 22815 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.191479 22815 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.193643 22815 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.197507 23138 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.198035 23138 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:00.186595 22901 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.199641 20467 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:00.202214 20616 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:00.203265 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 169453us
I20250831 13:59:00.200014 20498 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:00.205209 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:00.206035 20497 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:00.208580 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:00.208539 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:00.211928 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 46454.9us
I20250831 13:59:00.213284 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.214665 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:00.215312 20497 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:00.190177 22799 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.217204 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:00.217482 20502 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250831 13:59:00.217849 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:00.234454 20496 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:00.218660 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:00.220541 22819 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:59:00.200250 23138 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:00.223340 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:00.223968 23119 log.cc:927] Time spent T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.311s user 0.005s sys 0.001s
I20250831 13:59:00.224717 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.225005 22987 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.211572 23022 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:00.228967 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.144654 23000 raft_consensus.cc:738] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:00.213917 20571 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
W20250831 13:59:00.232656 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.318205 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:00.248601 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.320441 20643 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:59:00.322454 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:00.271174 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.311851 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:00.313264 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:00.251233 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.318346 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:00.325527 20573 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:00.201601 22901 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:00.319985 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:00.327474 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.235744 20497 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:59:00.323165 20570 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
W20250831 13:59:00.232403 22804 consensus_meta.cc:287] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.738s user 0.012s sys 0.001s
I20250831 13:59:00.203053 22815 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:00.326743 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 100644us
I20250831 13:59:00.215036 22369 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.332046 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.217479 23067 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:00.218035 23151 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.249611 22780 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:59:00.347944 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.318835 23022 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:00.373656 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.190629 22999 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.250033 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:00.349869 20616 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.351038 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:00.311861 22859 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
W20250831 13:59:00.357534 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 358289us
I20250831 13:59:00.219959 21176 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:59:00.378329 20571 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
W20250831 13:59:00.408232 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 58471.8us
I20250831 13:59:00.349642 23013 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.447454 20617 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.430023 20573 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:59:00.435604 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.448596 20616 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.437036 20569 raft_consensus.cc:1238] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:00.379065 22613 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:00.414947 20572 raft_consensus.cc:1273] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:59:00.407903 22804 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:00.381071 23022 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:00.442471 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:00.407054 22900 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:00.452301 20543 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.349373 22815 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.456298 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1396) took 2678 ms. Trace:
I20250831 13:59:00.474516 20568 rpcz_store.cc:276] 0831 13:58:57.778108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.075990 (+1297882us) service_pool.cc:225] Handling call
0831 13:58:59.076386 (+ 396us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:58:59.101900 (+ 25514us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:58:59.101961 (+ 61us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:58:59.115979 (+ 14018us) log.cc:844] Serialized 52 byte log entry
0831 13:58:59.739577 (+623598us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:58:59.739631 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:58:59.739698 (+ 67us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:00.223375 (+483677us) spinlock_profiling.cc:243] Waited 7.17 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:00.450745 (+227370us) raft_consensus.cc:1683] finished
0831 13:59:00.450771 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:00.456230 (+ 5459us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3150,"raft.queue_time_us":64923,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":16497408,"thread_start_us":584396,"threads_started":1,"wal-append.queue_time_us":694267}
I20250831 13:59:00.332815 23083 consensus_queue.cc:237] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.356122 22369 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.480311 23077 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.480787 23077 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.481503 20570 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 10.
I20250831 13:59:00.481725 22369 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.483492 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.483979 22815 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.485550 22804 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:59:00.486029 22804 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:00.486446 22804 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
W20250831 13:59:00.487329 23144 log.cc:927] Time spent T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
W20250831 13:59:00.487955 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.490052 20468 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:00.494053 22613 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:00.379681 23151 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.494812 22229 raft_consensus.cc:2802] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:00.495435 22229 raft_consensus.cc:695] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:00.496012 22999 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.496301 23151 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.495544 20617 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.453701 23013 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.339807 22850 consensus_queue.cc:260] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.504659 20617 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.505823 20467 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:59:00.510774 23045 consensus_meta.cc:287] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.627s user 0.007s sys 0.002s
I20250831 13:59:00.475530 23107 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:00.450768 22859 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:00.511886 23003 consensus_meta.cc:287] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.740s user 0.009s sys 0.004s
I20250831 13:59:00.518667 20617 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.421664 23111 consensus_queue.cc:237] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.494856 23156 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:59:00.535086 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 95849.7us
I20250831 13:59:00.537988 20466 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:00.522364 20574 raft_consensus.cc:2408] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250831 13:59:00.549691 20543 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:00.523995 20467 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:59:00.517793 22371 consensus_meta.cc:287] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.584s user 0.011s sys 0.000s
I20250831 13:59:00.511323 22999 raft_consensus.cc:2747] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:00.512043 23077 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.531968 23154 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:59:00.562779 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 79355.9us
I20250831 13:59:00.450333 23000 consensus_queue.cc:260] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:00.531569 22987 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.588773 20543 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:00.512115 22900 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.545148 22815 raft_consensus.cc:2802] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
W20250831 13:59:00.600054 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 76161.6us
I20250831 13:59:00.603112 20467 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.563477 23003 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:00.585467 22819 raft_consensus.cc:695] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:00.531315 23045 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.561878 22371 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.607757 22229 consensus_queue.cc:237] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.608544 22959 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:59:00.561031 22780 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:00.652236 22780 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:00.652933 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41404 (request call id 1511) took 1174 ms. Trace:
I20250831 13:59:00.653309 20576 rpcz_store.cc:276] 0831 13:58:59.478463 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.478575 (+ 112us) service_pool.cc:225] Handling call
0831 13:59:00.652769 (+1174194us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b54003b3b54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:00.652876 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1392191232}
I20250831 13:59:00.654263 22780 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:00.656708 22859 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.656571 22804 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.610846 23003 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:00.657578 23003 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:59:00.658452 23147 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250831 13:59:00.617452 23154 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:00.643970 22786 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.624162 23141 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.660975 22786 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.659209 22819 consensus_queue.cc:237] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.554250 23107 raft_consensus.cc:695] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 7 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:59:00.588848 23077 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:00.610396 22999 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:00.619032 20543 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:00.516060 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 293832us
I20250831 13:59:00.587612 22799 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.534080 23022 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.673583 23022 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.594000 23153 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:00.667100 22613 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 13:59:00.628177 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.675498 22613 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:00.616024 22815 raft_consensus.cc:695] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:00.650911 22959 raft_consensus.cc:695] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:00.675961 22613 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:00.676506 22815 consensus_queue.cc:237] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.676685 22959 consensus_queue.cc:237] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.605782 22767 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:00.561712 22850 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:00.656994 23067 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 13:59:00.688297 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 69373us
W20250831 13:59:00.628649 22873 consensus_meta.cc:287] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.568s user 0.009s sys 0.001s
I20250831 13:59:00.661234 23141 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.670293 23155 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20250831 13:59:00.668176 22804 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.646064 20467 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:00.614590 23000 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:00.681699 22767 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:00.707979 22338 consensus_queue.cc:1035] T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250831 13:59:00.713599 23160 log.cc:927] Time spent T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20250831 13:59:00.715824 23045 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.628043 22900 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:00.741559 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 95612.7us
I20250831 13:59:00.681710 23153 raft_consensus.cc:695] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:00.754428 20467 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:00.133536 20577 rpcz_store.cc:276] 0831 13:58:57.766291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:59.059359 (+1293068us) service_pool.cc:225] Handling call
0831 13:58:59.395148 (+335789us) spinlock_profiling.cc:243] Waited 6.28 ms on lock 0x7b58000f0398. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:58:59.395384 (+ 236us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:58:59.395667 (+ 283us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 13:58:59.783781 (+388114us) log.cc:844] Serialized 14 byte log entry
0831 13:59:00.101748 (+317967us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:00.101814 (+ 66us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:00.101861 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:00.101920 (+ 59us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:00.102069 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":303575,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":137866,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":22723840,"thread_start_us":662555,"threads_started":3,"wal-append.queue_time_us":219167}
I20250831 13:59:00.717619 23159 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:00.733297 22767 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:00.679860 22999 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.714576 23003 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.770905 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:00.775839 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.778391 20573 raft_consensus.cc:2391] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:59:00.778823 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.743556 22900 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:59:00.780316 20642 raft_consensus.cc:1273] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250831 13:59:00.782081 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 228417us
I20250831 13:59:00.784271 20643 raft_consensus.cc:1273] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:59:00.763069 23022 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:00.756686 23045 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:00.786854 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:00.787789 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.765925 23154 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election won for term 10
I20250831 13:59:00.789134 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:00.719308 22873 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:00.789535 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:00.794297 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.790633 20570 raft_consensus.cc:1273] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:59:00.794579 20648 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:00.791241 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:00.795343 20571 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:00.754205 23141 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:00.769502 23151 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.766898 22979 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:00.792835 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.778014 22660 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.755991 23111 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20250831 13:59:00.800580 22371 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.802933 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:00.804878 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:59:00.806473 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 89004.7us
I20250831 13:59:00.807742 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:00.808234 20643 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:59:00.812966 20543 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.763054 23132 consensus_queue.cc:1035] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20250831 13:59:00.816208 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:00.818590 22229 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250831 13:59:00.821116 20646 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:00.825361 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:00.825942 20649 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:59:00.828564 21206 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
W20250831 13:59:00.829532 23152 log.cc:927] Time spent T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250831 13:59:00.831804 20616 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.833709 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.834457 22901 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:00.835489 22780 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.836005 22780 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.838042 22780 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.840956 23138 consensus_queue.cc:1046] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:59:00.841243 22873 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.842921 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:00.845748 22901 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:00.847292 22999 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.796728 23022 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.848224 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:00.842219 23159 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:00.817353 23154 raft_consensus.cc:695] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 10 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:00.853246 20649 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:59:00.854918 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:00.894737 20572 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:00.856163 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:00.817759 22979 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.847054 23166 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 13:59:00.859503 23157 log.cc:927] Time spent T a05a337dd9e04dec9e8874b1d6c4d114 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.229s user 0.003s sys 0.000s
I20250831 13:59:00.859197 23153 consensus_queue.cc:237] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.781919 23164 consensus_queue.cc:1046] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.048s
I20250831 13:59:00.861315 23058 raft_consensus.cc:3053] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:00.864223 23003 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.860952 20617 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:00.870033 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 57189.9us
I20250831 13:59:00.830461 22613 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:00.872606 22804 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20250831 13:59:00.867746 22371 consensus_queue.cc:1035] T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20250831 13:59:00.939545 23132 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250831 13:59:00.947098 20501 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:59:00.875317 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:00.948351 20541 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:00.947134 23132 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:00.949122 23132 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:59:00.872298 20616 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.949752 23109 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:59:00.950264 23109 raft_consensus.cc:695] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
W20250831 13:59:00.882323 23133 log.cc:927] Time spent T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.775s user 0.003s sys 0.006s
I20250831 13:59:00.951388 23022 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.806871 23045 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.814738 22660 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.952978 20571 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:00.953231 23045 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.886873 20648 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 2.
I20250831 13:59:00.953912 22660 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.954946 23149 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:00.955494 23149 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:00.801543 23141 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:00.921358 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 123258us
I20250831 13:59:00.957548 23149 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.822018 22959 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:00.959384 23154 consensus_queue.cc:237] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.960747 20467 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.807034 23151 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.962627 20575 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 6
W20250831 13:59:00.930615 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 127640us
I20250831 13:59:00.964323 20576 raft_consensus.cc:1273] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:59:00.939018 20543 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.928937 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:00.941579 23013 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:00.928691 23058 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:00.946398 23177 raft_consensus.cc:3053] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:00.778282 23107 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:00.967015 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.985148 20568 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:59:00.986377 20467 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.967489 20541 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:00.982075 22900 raft_consensus.cc:695] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:00.968955 20571 raft_consensus.cc:1273] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250831 13:59:00.969671 23151 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.989806 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:00.990540 20648 raft_consensus.cc:2391] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:00.969833 23109 consensus_queue.cc:237] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.970615 22979 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:00.971223 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:00.971333 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.000155 20568 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:59:00.973485 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:00.973796 23180 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
W20250831 13:59:00.974850 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 102613us
W20250831 13:59:00.929486 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 68614.8us
I20250831 13:59:00.979014 20572 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:59:00.982087 20577 raft_consensus.cc:1273] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:59:00.948505 22613 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:00.984220 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.010106 20576 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
I20250831 13:59:00.987674 23138 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.989362 22987 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:00.987902 22999 raft_consensus.cc:2747] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:00.991977 23075 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:00.987422 22554 consensus_queue.cc:1035] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20250831 13:59:00.988874 22767 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.002511 20570 raft_consensus.cc:2391] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:01.007409 20616 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:00.997692 23141 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.006464 20617 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:00.981675 23177 raft_consensus.cc:738] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:00.988415 22660 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:00.996744 23132 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:00.989282 23159 raft_consensus.cc:2802] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250831 13:59:00.967571 22786 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
W20250831 13:59:01.023064 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 55688.7us
I20250831 13:59:01.004859 23111 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:01.031141 20574 raft_consensus.cc:1273] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250831 13:59:01.031759 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:01.020314 22767 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:01.042151 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.015911 23075 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:01.031947 21208 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 6: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
I20250831 13:59:01.037077 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.047087 20649 raft_consensus.cc:2391] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:01.043118 20496 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:01.036270 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.049487 20643 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:01.051055 23158 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250831 13:59:01.051355 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.034368 23022 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:01.063158 23022 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.064203 23132 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:01.065168 23022 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.024680 22660 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:01.023310 23180 raft_consensus.cc:695] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:01.064754 23177 consensus_queue.cc:260] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.015849 22987 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.088585 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.089293 20573 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:01.089696 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.090227 20572 raft_consensus.cc:3058] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:59:01.090885 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44457.4us
I20250831 13:59:01.014701 23138 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.081223 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.081491 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.065933 22660 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:59:01.066493 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 69340.7us
I20250831 13:59:01.087263 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:01.096931 20646 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:01.035734 23159 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:01.088515 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.092124 20568 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250831 13:59:01.097560 20498 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:01.098718 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:01.098989 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:01.099750 20568 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:01.100980 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.036396 22371 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:59:01.093149 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.094952 20576 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:01.056666 23075 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:01.033252 23058 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:01.110206 20571 raft_consensus.cc:3058] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:01.066180 23164 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20250831 13:59:01.115603 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.119738 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.036743 22815 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:01.119163 20493 raft_consensus.cc:1273] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20250831 13:59:01.082618 22959 consensus_queue.cc:1046] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20250831 13:59:01.082566 22767 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:01.133672 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.135167 20577 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
W20250831 13:59:01.145761 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 48916us
I20250831 13:59:01.082362 22979 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:01.149101 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.139271 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.140177 20499 raft_consensus.cc:2391] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 4: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:01.120512 20497 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:01.150002 20576 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:59:01.139256 22999 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:01.043578 23111 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.155157 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.155768 20568 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:01.156265 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.156618 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.157310 20568 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:01.158332 20573 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:01.151830 20648 raft_consensus.cc:1273] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:59:01.151619 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.159107 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.151679 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:01.143803 23179 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20250831 13:59:01.107779 23159 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:59:01.107476 23127 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20250831 13:59:01.170013 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.139200 23158 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:01.189694 20574 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:01.172319 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.118845 22371 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:01.135030 23177 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:01.173455 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.190546 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.200850 20499 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:01.200842 20577 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:01.202175 20467 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:01.240479 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.152186 22815 raft_consensus.cc:695] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:01.245713 20648 raft_consensus.cc:1273] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:59:01.249603 20571 raft_consensus.cc:2466] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:01.250504 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:01.250840 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 48739us
I20250831 13:59:01.251087 20571 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:59:01.243286 20568 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:01.080276 22900 consensus_queue.cc:237] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.256596 20568 raft_consensus.cc:1273] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20250831 13:59:01.255136 20543 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.262223 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.262883 20493 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:01.263222 20617 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:01.262406 20568 pending_rounds.cc:85] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 2
I20250831 13:59:01.264999 20568 pending_rounds.cc:107] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20250831 13:59:01.265358 20568 raft_consensus.cc:2887] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:01.151440 22767 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.268747 20572 raft_consensus.cc:2466] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:01.269733 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:01.273649 20648 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:01.270902 20543 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.156988 22979 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:59:01.154716 22999 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.197579 22786 consensus_queue.cc:1035] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20250831 13:59:01.275697 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.282407 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.277467 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.274564 20572 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:01.276913 20468 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:01.254922 22371 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:59:01.250735 22859 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:01.255420 22882 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:01.332013 20643 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
W20250831 13:59:01.345737 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 68920.6us
I20250831 13:59:01.349504 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.276598 23183 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:01.300457 20495 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:59:01.350837 20568 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:01.352104 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:01.278914 23000 consensus_meta.cc:287] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.538s user 0.010s sys 0.000s
I20250831 13:59:01.353695 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.357622 20575 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
W20250831 13:59:01.358556 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 87731.4us
I20250831 13:59:01.276844 23013 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 13:59:01.323562 23181 log.cc:927] Time spent T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.265s user 0.001s sys 0.000s
I20250831 13:59:01.242223 23158 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:01.370309 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:01.372641 20500 raft_consensus.cc:1273] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:01.374838 20568 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20250831 13:59:01.380112 20493 raft_consensus.cc:2466] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:01.343422 22554 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:59:01.395562 20643 raft_consensus.cc:1273] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250831 13:59:01.373354 23168 log.cc:927] Time spent T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.607s user 0.002s sys 0.007s
I20250831 13:59:01.352943 23183 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:01.278353 22369 consensus_queue.cc:1046] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20250831 13:59:01.298880 23003 consensus_queue.cc:1035] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20250831 13:59:01.374123 23126 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:01.368875 23013 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:01.411206 20617 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:01.251776 23180 consensus_queue.cc:237] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.416671 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.430887 20649 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:01.355093 22859 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.360178 23127 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250831 13:59:01.360111 22882 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.442548 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.410504 23138 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.425248 22613 consensus_queue.cc:1035] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20250831 13:59:01.430339 23013 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:01.404639 22987 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.437927 20642 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:01.420830 23126 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:01.409474 23183 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:59:01.461501 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 50412.7us
I20250831 13:59:01.485672 20541 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:59:01.491741 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 117775us
W20250831 13:59:01.494741 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:01.495452 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.495783 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.430653 23125 consensus_queue.cc:1035] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20250831 13:59:01.496596 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.494534 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.500797 20496 raft_consensus.cc:2408] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20250831 13:59:01.453518 23111 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.472236 23126 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:01.504233 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:01.483551 23188 log.cc:927] Time spent T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.336s user 0.000s sys 0.005s
I20250831 13:59:01.513523 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.513680 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.399729 22815 consensus_queue.cc:237] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.479609 23058 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:01.517314 20496 raft_consensus.cc:2391] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:01.484701 23138 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250831 13:59:01.516220 20642 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:01.472765 23200 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:01.525049 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.524159 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.564986 20500 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:59:01.565667 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.567512 20496 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:01.570398 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.524032 23083 raft_consensus.cc:2802] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:01.578179 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.577852 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.583984 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.584837 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.586177 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.537461 20495 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 11.
I20250831 13:59:01.587759 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.538151 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.589154 20575 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Advancing to term 11
W20250831 13:59:01.566146 23186 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.298s user 0.001s sys 0.000s
I20250831 13:59:01.540818 20573 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20250831 13:59:01.524654 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.522647 20646 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:01.542356 20499 raft_consensus.cc:2466] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:01.563925 20648 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:01.596688 20573 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:01.598379 20573 raft_consensus.cc:1273] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250831 13:59:01.532542 22882 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:01.539083 23187 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.294s user 0.000s sys 0.002s
W20250831 13:59:01.562091 22850 consensus_meta.cc:287] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.859s user 0.004s sys 0.006s
I20250831 13:59:01.525445 21206 raft_consensus.cc:1273] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20250831 13:59:01.602097 20616 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.607947 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.608927 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.588465 20495 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:01.569126 22999 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.534194 22987 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:01.650286 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.614151 20575 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 11.
I20250831 13:59:01.603461 20494 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:01.562367 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:01.629020 20496 raft_consensus.cc:2366] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:59:01.591647 22767 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:01.592775 23158 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.629938 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.557168 22979 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:59:01.526669 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:59:01.537329 20447 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 90a0a56228d04660b3cb351485be6593 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ab7ade51fa134a1b814fab748f01d43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } } }
I20250831 13:59:01.629544 22959 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:01.561551 23075 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:01.671670 23186 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20250831 13:59:01.534257 23173 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.211s
I20250831 13:59:01.677838 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.675195 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.705269 20499 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:59:01.686503 20493 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 7. Ops: [5.4-5.4]
I20250831 13:59:01.600865 23083 raft_consensus.cc:695] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:59:01.717607 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 298985us
I20250831 13:59:01.694145 22959 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:01.719192 20466 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:59:01.716571 20498 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:01.672667 22987 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:59:01.705945 22859 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:01.707865 20447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 170587us
I20250831 13:59:01.729753 20466 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:01.715627 23210 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:01.711864 22767 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20250831 13:59:01.736862 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.747864 20493 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 11.
I20250831 13:59:01.719333 23158 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:01.720249 22959 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:01.779485 23075 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.780287 23183 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.780647 22804 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250831 13:59:01.782131 20466 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:01.783282 20466 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:01.783352 23162 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:01.783686 23183 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.784129 23162 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:01.787591 22987 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:01.737056 23184 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:01.733973 22853 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 13:59:01.755757 23206 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20250831 13:59:01.746097 23210 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:59:01.764055 23159 consensus_meta.cc:287] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.550s user 0.007s sys 0.002s
I20250831 13:59:01.764598 23083 consensus_queue.cc:237] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.790526 23126 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.745165 20500 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
W20250831 13:59:01.770107 22660 consensus_meta.cc:287] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.648s user 0.009s sys 0.002s
I20250831 13:59:01.749711 23106 raft_consensus.cc:3053] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
W20250831 13:59:01.798509 20502 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s user 0.011s sys 0.022s
W20250831 13:59:01.804517 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 202514us
I20250831 13:59:01.808135 20467 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:01.792848 23141 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:01.852664 20616 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.796548 22853 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:01.776026 23158 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.858003 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.858752 20575 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:01.859375 20500 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20250831 13:59:01.844354 20502 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.546s user 0.017s sys 0.022s
I20250831 13:59:01.798979 23210 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:01.869297 20569 raft_consensus.cc:1273] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:59:01.784384 22371 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.797034 22882 consensus_queue.cc:1035] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20250831 13:59:01.809397 23106 raft_consensus.cc:738] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:01.876624 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.804425 23075 consensus_queue.cc:1046] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250831 13:59:01.798322 23127 consensus_queue.cc:1035] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20250831 13:59:01.874106 23153 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:01.878157 23205 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:01.895316 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.895898 20577 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 11.
I20250831 13:59:01.879590 20569 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:01.808907 23184 consensus_queue.cc:1035] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20250831 13:59:01.855325 23141 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.803117 23159 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.858916 22853 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:01.902608 20651 raft_consensus.cc:2391] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:01.860396 22900 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:59:01.888430 23107 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 13:59:01.894943 23205 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:01.907399 22371 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:01.892865 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:01.909291 23205 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:01.900627 23153 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:01.893154 23162 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:59:01.911123 23177 consensus_meta.cc:287] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.669s user 0.008s sys 0.002s
I20250831 13:59:01.911397 23153 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:01.911852 20497 raft_consensus.cc:2408] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20250831 13:59:01.881103 23219 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:59:01.912631 20573 raft_consensus.cc:1238] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:59:01.912479 23210 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:01.881737 20543 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.899555 20493 raft_consensus.cc:1238] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:01.812182 22660 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:01.884799 23185 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.644s user 0.001s sys 0.003s
I20250831 13:59:01.902937 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:01.918905 20543 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.919175 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:01.918444 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 65860us
I20250831 13:59:01.920317 20543 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:01.917851 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 182366us
I20250831 13:59:01.922140 20543 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.931143 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:01.931425 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.931859 20648 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:01.935518 20617 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:01.939778 20543 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.941452 20543 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.947417 20543 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.960043 20617 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:01.961769 20543 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.964818 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.966272 20543 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.969472 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.974556 20645 raft_consensus.cc:1273] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20250831 13:59:01.974982 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:01.975528 20577 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:01.952989 20616 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.978297 20650 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 13:59:01.977092 20617 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:01.980757 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.933369 22900 raft_consensus.cc:695] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:01.987757 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:01.995272 20543 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.996783 20616 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.999277 20467 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:02.003674 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:01.946508 23205 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:01.956050 23109 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:59:01.960804 23197 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:59:02.004280 20647 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 9
I20250831 13:59:01.945981 23153 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:59:02.005491 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 61014.4us
I20250831 13:59:02.005290 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.959394 22850 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Leader election won for term 11
I20250831 13:59:02.024406 20646 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 8.
I20250831 13:59:02.025086 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.944759 22371 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20250831 13:59:01.913965 20571 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992"
dest_uuid: "212b66976f874aa682cae0fb77928786"
mode: GRACEFUL
new_leader_uuid: "58aded50ab0d4609a053647eb11b1f48"
from {username='slave'} at 127.0.0.1:49676
I20250831 13:59:01.929356 22927 consensus_queue.cc:1046] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20250831 13:59:02.034049 20571 raft_consensus.cc:604] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Received request to transfer leadership to TS 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:01.954777 23200 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:02.034554 20571 raft_consensus.cc:612] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:59:02.016217 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:02.035419 20575 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:02.017915 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:02.018500 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.966354 22780 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:59:02.019722 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 62040.2us
I20250831 13:59:02.019986 20648 raft_consensus.cc:1273] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250831 13:59:01.998584 22999 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:02.004762 23132 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250831 13:59:01.982159 22979 raft_consensus.cc:2747] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:02.026391 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:02.027659 20543 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:02.013861 20577 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:02.020718 23167 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:01.944974 23223 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250831 13:59:01.945899 23213 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:02.039108 20496 raft_consensus.cc:1273] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20250831 13:59:02.039532 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:02.041268 22660 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:02.041857 23126 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:02.048044 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:01.933516 23106 consensus_queue.cc:260] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.049427 20498 raft_consensus.cc:1273] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:59:02.050091 22853 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.050556 20571 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250831 13:59:02.054405 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:02.056301 20647 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:02.089859 20494 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:59:02.052423 23205 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:02.056514 20574 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:02.017227 23173 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250831 13:59:01.988119 22767 raft_consensus.cc:2802] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:02.061650 20577 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250831 13:59:02.063709 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:01.982040 23077 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20250831 13:59:02.022864 23138 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:02.080538 20573 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 11.
I20250831 13:59:02.005254 21174 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:02.048636 23224 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:02.038578 23210 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:02.087821 20500 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:02.092523 20649 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 11.
I20250831 13:59:02.061049 23058 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:59:02.095955 20647 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 9 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:02.095088 20642 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:02.061954 23167 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:02.063877 22850 raft_consensus.cc:695] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 11 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:02.050968 22780 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:02.084021 23013 consensus_meta.cc:287] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.597s user 0.009s sys 0.001s
I20250831 13:59:02.032465 23151 consensus_queue.cc:1046] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20250831 13:59:02.102766 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:02.034258 23182 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
W20250831 13:59:02.105894 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 52483us
W20250831 13:59:02.109159 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 49711.2us
I20250831 13:59:02.141659 20616 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:02.115986 23173 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:02.057359 23200 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.118590 20642 raft_consensus.cc:1238] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:02.118875 20647 raft_consensus.cc:1273] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:59:02.146914 20466 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:02.113930 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 86346.9us
I20250831 13:59:02.147838 23126 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:02.148321 23126 raft_consensus.cc:695] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 9 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:02.148401 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:02.125363 20648 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:02.149184 23126 consensus_queue.cc:237] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.098308 23158 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:02.100374 23106 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:02.089963 23127 consensus_queue.cc:1046] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20250831 13:59:02.116528 20494 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:59:02.116644 22767 raft_consensus.cc:695] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:02.126541 23167 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:02.154958 23215 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20250831 13:59:02.121767 23045 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:59:02.098613 23083 consensus_queue.cc:1046] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.033s
I20250831 13:59:02.126942 23107 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:59:02.142107 22660 consensus_queue.cc:1035] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20250831 13:59:02.164569 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:02.167811 22780 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:02.153805 22767 consensus_queue.cc:237] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.171813 22853 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:02.113797 23232 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:02.160600 20574 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:02.116343 23224 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.163251 22873 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:02.176431 20616 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:02.176957 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:02.120368 23132 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250831 13:59:02.107707 23138 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.109432 23225 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.052s
I20250831 13:59:02.188750 20616 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:02.183707 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:02.091180 22900 consensus_queue.cc:237] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.131237 23231 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:59:02.192081 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 168596us
I20250831 13:59:02.146742 23173 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Advancing to term 12
I20250831 13:59:02.199142 20467 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:02.204491 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:02.205111 20643 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:59:02.145987 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 171825us
I20250831 13:59:02.180500 22979 raft_consensus.cc:3053] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:02.140069 23013 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:02.223711 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 75390.7us
W20250831 13:59:02.223834 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 47422.8us
I20250831 13:59:02.224663 20543 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:02.225473 20467 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:02.215271 22613 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250831 13:59:02.216295 23154 raft_consensus.cc:3053] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:02.217425 20647 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20250831 13:59:02.222301 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 96321.9us
I20250831 13:59:02.187249 22371 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:02.144124 23205 consensus_queue.cc:1046] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20250831 13:59:02.173646 23125 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20250831 13:59:02.258301 20543 leader_election.cc:400] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:59:02.250715 20467 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:02.203758 23000 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:02.206379 23210 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:59:02.258785 20543 leader_election.cc:403] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20250831 13:59:02.182592 22999 consensus_queue.cc:1035] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20250831 13:59:02.212260 22979 raft_consensus.cc:738] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:02.181996 22959 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.244217 22613 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:02.181829 22873 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:02.275707 20573 raft_consensus.cc:1273] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:59:02.284514 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:02.253463 23045 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:59:02.253394 22371 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:02.248431 23154 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:02.279341 23209 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:02.279660 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:02.297434 20575 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:59:02.279840 23220 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20250831 13:59:02.281221 22959 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:02.282904 20569 raft_consensus.cc:1238] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250831 13:59:02.293527 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:02.285578 20543 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:02.225626 22850 consensus_queue.cc:237] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.223431 23217 consensus_queue.cc:1035] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20250831 13:59:02.301870 23245 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:02.247926 23219 consensus_queue.cc:1035] T ca1477952e8c40adb14c71db5a7a4b92 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20250831 13:59:02.302441 23245 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:02.302983 23245 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:02.283463 22613 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Advancing to term 12
I20250831 13:59:02.300577 20617 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:02.278322 23205 consensus_queue.cc:1230] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20250831 13:59:02.304131 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:02.309890 20645 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 8: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:02.307124 20543 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:02.292295 23125 raft_consensus.cc:2747] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:02.312215 20577 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:02.284381 23000 raft_consensus.cc:695] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:02.294775 22371 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:02.320856 20574 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250831 13:59:02.260612 23216 consensus_queue.cc:1046] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20250831 13:59:02.291869 23164 consensus_queue.cc:1046] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20250831 13:59:02.297158 23209 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:59:02.343181 23235 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20250831 13:59:02.338748 23257 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:02.303957 23229 consensus_queue.cc:1035] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20250831 13:59:02.420329 20643 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:02.420538 20498 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:02.340097 23222 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6.
I20250831 13:59:02.350720 20642 raft_consensus.cc:1238] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 13:59:02.351941 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:02.353416 20494 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250831 13:59:02.422715 20575 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:59:02.353788 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 62535.7us
I20250831 13:59:02.368407 21174 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:59:02.369374 23058 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:02.369827 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:02.311491 23132 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election won for term 5
I20250831 13:59:02.326004 23205 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:02.370301 23164 raft_consensus.cc:3053] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:02.357637 23003 raft_consensus.cc:3053] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:02.265208 22767 consensus_queue.cc:1035] T 5040bfaecba143f4af6334f314734a46 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
W20250831 13:59:02.500208 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 148337us
I20250831 13:59:02.419394 23224 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:02.503181 22873 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:02.375928 23209 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:59:02.509876 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 84180.4us
I20250831 13:59:02.426605 23210 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:02.313095 22979 consensus_queue.cc:260] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:02.636085 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 253552us
W20250831 13:59:02.653381 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 282060us
I20250831 13:59:02.597985 23003 raft_consensus.cc:738] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:02.630780 23164 raft_consensus.cc:738] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:02.656764 23246 consensus_queue.cc:1035] T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250831 13:59:02.658718 23241 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.354s user 0.000s sys 0.000s
W20250831 13:59:02.659869 23236 log.cc:927] Time spent T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.401s user 0.000s sys 0.001s
I20250831 13:59:02.425841 22804 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:02.426777 23138 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:02.425144 23257 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.513856 23106 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:59:02.663637 23225 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:02.381279 23167 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.665630 23000 consensus_queue.cc:237] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.429399 23231 raft_consensus.cc:2762] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20250831 13:59:02.701884 20467 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:59:02.705252 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 106610us
I20250831 13:59:02.357033 23154 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.700930 23013 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:02.835075 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:02.569481 23058 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.846604 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1591) took 1122 ms. Trace:
I20250831 13:59:02.802053 23203 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:02.843497 23173 consensus_meta.cc:287] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.632s user 0.010s sys 0.000s
I20250831 13:59:02.851292 20499 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:02.801793 23132 raft_consensus.cc:695] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:02.856439 20501 rpcz_store.cc:276] 0831 13:59:01.724571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.724711 (+ 140us) service_pool.cc:225] Handling call
0831 13:59:01.725024 (+ 313us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:01.742619 (+ 17595us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:01.742663 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:01.742738 (+ 75us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:01.742786 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:02.846313 (+1103527us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b540021c0d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:02.846375 (+ 62us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:02.846538 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39936256}
I20250831 13:59:02.792131 23210 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:02.791246 23045 consensus_queue.cc:1046] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20250831 13:59:02.832310 23125 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:02.836347 23225 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:02.901485 20493 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:02.836930 22804 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:03.452802 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 750989us
I20250831 13:59:02.987989 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37606 (request call id 78) took 1073 ms. Trace:
W20250831 13:59:03.373679 23250 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20250831 13:59:03.472836 20495 rpcz_store.cc:276] 0831 13:59:01.914166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.914391 (+ 225us) service_pool.cc:225] Handling call
0831 13:59:01.917803 (+ 3412us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:01.934460 (+ 16657us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:01.934502 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:01.937579 (+ 3077us) log.cc:844] Serialized 26 byte log entry
0831 13:59:02.099036 (+161457us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:59:02.099080 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:02.099148 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:02.987693 (+888545us) raft_consensus.cc:1683] finished
0831 13:59:02.987724 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:02.987909 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58061,"raft.queue_time_us":20308,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":10092,"thread_start_us":118179,"threads_started":1,"wal-append.queue_time_us":302702}
W20250831 13:59:03.472376 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 1.104e+06us
W20250831 13:59:02.960170 23237 log.cc:927] Time spent T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20250831 13:59:03.479146 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:02.988256 23003 consensus_queue.cc:260] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:03.473282 23261 log.cc:927] Time spent T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.439s user 0.009s sys 0.004s
I20250831 13:59:03.291785 22959 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:03.409073 23225 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:03.494630 22369 consensus_queue.cc:1046] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20250831 13:59:03.497879 22660 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:03.040678 23256 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.196s user 0.000s sys 0.000s
W20250831 13:59:02.959962 23226 log.cc:927] Time spent T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20250831 13:59:03.498453 22660 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:03.495215 23262 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.575s user 0.001s sys 0.000s
I20250831 13:59:03.499816 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1478) took 2163 ms. Trace:
W20250831 13:59:03.040865 23176 log.cc:927] Time spent T 00000000000000000000000000000000 P 6f9e84d5d5f24202bf6812740505bbfd: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
I20250831 13:59:03.500445 22660 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:03.495141 23151 raft_consensus.cc:3053] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:03.501029 23167 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:03.497152 22959 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.500203 20568 rpcz_store.cc:276] 0831 13:59:01.336567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.383119 (+ 46552us) service_pool.cc:225] Handling call
0831 13:59:01.383478 (+ 359us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:01.595368 (+211890us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:01.595401 (+ 33us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:01.600150 (+ 4749us) log.cc:844] Serialized 26 byte log entry
0831 13:59:01.951736 (+351586us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:01.951784 (+ 48us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:01.951877 (+ 93us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:03.499599 (+1547722us) raft_consensus.cc:1683] finished
0831 13:59:03.499622 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:03.499746 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98146,"raft.queue_time_us":215585,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":14999,"thread_start_us":432964,"threads_started":2,"wal-append.queue_time_us":600584}
I20250831 13:59:03.504218 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:03.504457 22959 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:03.504746 20568 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:59:03.505014 22959 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:03.505527 22959 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.501046 23151 raft_consensus.cc:738] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:03.496203 20467 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:03.506729 23151 consensus_queue.cc:260] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.291522 23204 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:03.509081 23264 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:03.500691 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1561) took 1226 ms. Trace:
I20250831 13:59:03.509557 23264 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:03.510061 23264 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:03.496043 23003 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:03.512532 23182 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 13:59:03.509850 20571 rpcz_store.cc:276] 0831 13:59:02.274080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.276132 (+ 2052us) service_pool.cc:225] Handling call
0831 13:59:02.276399 (+ 267us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:02.286149 (+ 9750us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:02.286192 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:02.286697 (+ 505us) log.cc:844] Serialized 26 byte log entry
0831 13:59:02.398440 (+111743us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:02.398507 (+ 67us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:02.398598 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:03.500476 (+1101878us) raft_consensus.cc:1683] finished
0831 13:59:03.500502 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:03.500636 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9349,"raft.queue_time_us":10172,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":3886,"thread_start_us":75586,"threads_started":1,"wal-append.queue_time_us":501996}
I20250831 13:59:03.516206 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:03.517504 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:03.518188 20571 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:59:03.418668 23154 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:03.492349 23159 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:03.493257 23141 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:03.373349 22613 consensus_meta.cc:287] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.052s user 0.009s sys 0.000s
I20250831 13:59:02.910351 23106 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:03.516481 20499 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:03.523262 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:03.426819 22900 raft_consensus.cc:2747] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:59:03.038288 23058 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:03.508116 23204 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:03.530503 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.160s user 0.003s sys 0.011s
I20250831 13:59:03.498905 23166 raft_consensus.cc:3053] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
W20250831 13:59:03.592701 22850 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.289s user 0.005s sys 0.015s
W20250831 13:59:03.613417 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.243s user 0.003s sys 0.011s
W20250831 13:59:03.686276 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 1.31608e+06us
I20250831 13:59:03.687167 20616 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:03.688858 20616 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:03.502175 23167 consensus_queue.cc:1046] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20250831 13:59:02.922899 23173 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:03.742987 20468 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.906s user 0.022s sys 0.155s
W20250831 13:59:03.759425 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 49912.5us
W20250831 13:59:03.762853 20468 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.926s user 0.022s sys 0.155s
I20250831 13:59:02.916240 23203 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.600957 23159 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:03.692196 23253 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.907s user 0.000s sys 0.001s
I20250831 13:59:03.610982 23265 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
W20250831 13:59:03.684136 22850 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.380s user 0.005s sys 0.016s
I20250831 13:59:03.676852 23166 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:03.711885 22901 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:59:03.712174 23217 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:03.466658 23125 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.767066 22900 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:03.679559 23106 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.613000 22613 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:03.796000 20573 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.496s user 0.027s sys 0.017s
W20250831 13:59:03.795698 20575 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.885s user 0.000s sys 0.043s
I20250831 13:59:02.917734 23164 consensus_queue.cc:260] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.780634 22959 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:03.818845 22850 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:03.846779 20575 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.936s user 0.001s sys 0.044s
W20250831 13:59:03.847795 20573 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.548s user 0.028s sys 0.017s
I20250831 13:59:03.780530 23083 consensus_queue.cc:1046] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20250831 13:59:03.803556 22901 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:03.496115 23270 consensus_queue.cc:1162] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "58aded50ab0d4609a053647eb11b1f48" responder_term: 7 status { last_received { term: 6 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 13:59:03.778940 23159 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.759418 23132 consensus_queue.cc:237] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:03.876518 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:03.856494 22850 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.848918 23058 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:03.847237 23217 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:03.927944 20466 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.008s user 0.019s sys 0.054s
I20250831 13:59:03.853031 22959 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:03.855396 23265 raft_consensus.cc:695] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:03.918159 23167 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
W20250831 13:59:03.959348 23275 log.cc:927] Time spent T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
I20250831 13:59:03.961091 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1439) took 1611 ms. Trace:
I20250831 13:59:03.870599 22901 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.961479 20647 rpcz_store.cc:276] 0831 13:59:02.349515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.349621 (+ 106us) service_pool.cc:225] Handling call
0831 13:59:02.349913 (+ 292us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:02.422612 (+ 72699us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:02.422647 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:02.425671 (+ 3024us) log.cc:844] Serialized 26 byte log entry
0831 13:59:02.809098 (+383427us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:02.862406 (+ 53308us) log.cc:844] Serialized 14 byte log entry
0831 13:59:03.406274 (+543868us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:03.406374 (+ 100us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:03.960831 (+554457us) raft_consensus.cc:1683] finished
0831 13:59:03.960870 (+ 39us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:03.961016 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":169206,"prepare.queue_time_us":150191,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":490739,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":192,"thread_start_us":104495,"threads_started":1,"wal-append.queue_time_us":1090114}
I20250831 13:59:03.963706 23167 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:03.965909 23167 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:03.931865 23210 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:03.927965 23268 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:03.842293 22900 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:03.942926 20466 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.023s user 0.019s sys 0.054s
I20250831 13:59:03.929996 23257 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:03.972564 22850 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:03.966435 23167 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:03.988425 22371 consensus_meta.cc:287] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 1.631s user 0.010s sys 0.000s
I20250831 13:59:03.993712 23203 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:03.955042 23058 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:03.994376 23204 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:03.975857 22804 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:03.957872 22959 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:03.995904 22613 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:03.999380 23274 log.cc:927] Time spent T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20250831 13:59:04.000065 20645 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1106) took 1655 ms. Trace:
I20250831 13:59:03.952618 23217 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:04.008329 20541 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.000s sys 0.012s
W20250831 13:59:04.006532 20540 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.012s sys 0.037s
W20250831 13:59:04.022279 20541 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.000s sys 0.013s
W20250831 13:59:04.022352 20540 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s user 0.012s sys 0.037s
W20250831 13:59:04.036490 20615 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.174s user 0.012s sys 0.002s
I20250831 13:59:04.014523 20645 rpcz_store.cc:276] 0831 13:59:02.344695 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.346327 (+ 1632us) service_pool.cc:225] Handling call
0831 13:59:02.379851 (+ 33524us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:02.652619 (+272768us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:02.652667 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:02.773619 (+120952us) log.cc:844] Serialized 26 byte log entry
0831 13:59:03.251230 (+477611us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:59:03.251282 (+ 52us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:03.251374 (+ 92us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:03.999776 (+748402us) raft_consensus.cc:1683] finished
0831 13:59:03.999807 (+ 31us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:03.999982 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":133449,"raft.queue_time_us":284895,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":87,"thread_start_us":550242,"threads_started":2,"wal-append.queue_time_us":1038031}
I20250831 13:59:04.007769 23203 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:03.983631 23268 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.012058 23204 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.002234 23167 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:04.025056 20614 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.000s sys 0.013s
W20250831 13:59:04.052279 20614 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.554s user 0.000s sys 0.013s
W20250831 13:59:04.024030 20617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.000s sys 0.013s
W20250831 13:59:04.052978 20617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s user 0.001s sys 0.013s
I20250831 13:59:03.981427 20466 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:04.039549 20615 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.178s user 0.012s sys 0.002s
W20250831 13:59:04.010979 22369 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.005s sys 0.007s
I20250831 13:59:04.006547 22371 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:04.037845 23232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.010s sys 0.003s
W20250831 13:59:04.036633 20467 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.003s sys 0.012s
W20250831 13:59:04.066318 20467 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.003s sys 0.012s
W20250831 13:59:04.068871 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 87620.1us
I20250831 13:59:04.071974 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1569) took 1701 ms. Trace:
I20250831 13:59:04.050452 23125 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.076937 23058 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:04.080207 23289 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.080786 23289 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:04.081431 22613 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:04.082109 23284 consensus_queue.cc:1046] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 13:59:04.082991 21173 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.011s sys 0.078s
I20250831 13:59:04.083551 23289 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:04.088848 21173 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.011s sys 0.078s
I20250831 13:59:04.083772 23288 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:04.086407 23225 consensus_meta.cc:287] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Time spent flushing consensus metadata: real 0.589s user 0.008s sys 0.000s
I20250831 13:59:04.075028 20575 rpcz_store.cc:276] 0831 13:59:02.370881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.426911 (+ 56030us) service_pool.cc:225] Handling call
0831 13:59:02.506109 (+ 79198us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:02.707138 (+201029us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:02.707185 (+ 47us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:02.806142 (+ 98957us) log.cc:844] Serialized 26 byte log entry
0831 13:59:03.849561 (+1043419us) raft_consensus.cc:1643] Marking committed up to 8
0831 13:59:03.849604 (+ 43us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:03.849666 (+ 62us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:04.071700 (+222034us) raft_consensus.cc:1683] finished
0831 13:59:04.071730 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:04.071899 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":133491,"raft.queue_time_us":193459,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":67536,"thread_start_us":938825,"threads_started":1,"wal-append.queue_time_us":1123376}
W20250831 13:59:04.063591 22369 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.005s sys 0.007s
W20250831 13:59:04.099079 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:04.068511 23232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s user 0.010s sys 0.003s
I20250831 13:59:04.051102 23106 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:04.068699 23000 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s user 0.000s sys 0.011s
I20250831 13:59:04.054066 23203 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.103396 23269 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:59:04.103235 23289 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.103294 22613 raft_consensus.cc:695] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:04.106380 22959 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.108855 23284 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:04.125366 23106 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:04.119678 23000 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s user 0.001s sys 0.012s
W20250831 13:59:04.144025 20651 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 1093) took 2649 ms (client timeout 2454 ms). Trace:
I20250831 13:59:04.154158 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:04.126014 23278 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.243s user 0.000s sys 0.001s
I20250831 13:59:04.109655 23287 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.111689 23058 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:04.141131 23141 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.149087 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:59:04.126757 23245 consensus_meta.cc:287] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 1.823s user 0.008s sys 0.006s
I20250831 13:59:04.114939 23159 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.063728 23204 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:04.031630 23265 consensus_queue.cc:237] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:04.154557 20651 rpcz_store.cc:269] 0831 13:59:01.495002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.495475 (+ 473us) service_pool.cc:225] Handling call
0831 13:59:04.143818 (+2648343us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b540029fc54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:04.143950 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":887928576}
I20250831 13:59:04.102670 23225 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:04.105808 23288 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.169531 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57698 (request call id 1419) took 2201 ms. Trace:
I20250831 13:59:04.140805 23292 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.199719 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1565) took 1923 ms. Trace:
I20250831 13:59:04.193595 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1486) took 2328 ms. Trace:
W20250831 13:59:04.274458 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 204317us
I20250831 13:59:04.301924 20649 rpcz_store.cc:276] 0831 13:59:01.967705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.967822 (+ 117us) service_pool.cc:225] Handling call
0831 13:59:04.169325 (+2201503us) spinlock_profiling.cc:243] Waited 70.7 ms on lock 0x7b5400069854. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:04.169459 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162534144}
I20250831 13:59:04.260114 20499 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:59:04.300850 20568 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:04.308838 20497 raft_consensus.cc:1238] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:04.308101 20569 rpcz_store.cc:276] 0831 13:59:01.865495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.896629 (+ 31134us) service_pool.cc:225] Handling call
0831 13:59:01.896901 (+ 272us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:02.303165 (+406264us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:04.193312 (+1890147us) spinlock_profiling.cc:243] Waited 386 ms on lock 0x7b54002a99d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:04.193526 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":887221632}
I20250831 13:59:04.310813 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.311270 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.308445 20573 rpcz_store.cc:276] 0831 13:59:02.275884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.285300 (+ 9416us) service_pool.cc:225] Handling call
0831 13:59:02.285597 (+ 297us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:02.285795 (+ 198us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:02.285825 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:02.286201 (+ 376us) log.cc:844] Serialized 26 byte log entry
0831 13:59:03.850256 (+1564055us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:04.115673 (+265417us) log.cc:844] Serialized 14 byte log entry
0831 13:59:04.199327 (+ 83654us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:04.199426 (+ 99us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:04.199486 (+ 60us) raft_consensus.cc:1683] finished
0831 13:59:04.199505 (+ 19us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:04.199654 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":329014,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":1816,"raft.queue_time_us":80256,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":1898736,"threads_started":3,"wal-append.queue_time_us":1664201}
I20250831 13:59:04.314440 23232 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:04.194059 23154 consensus_meta.cc:287] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.665s user 0.006s sys 0.003s
I20250831 13:59:04.198462 23203 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.150099 22959 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.315804 22959 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.301733 23125 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:04.316918 23125 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.317760 22959 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.194793 22371 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:04.318572 22371 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:04.319056 22371 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.319550 23305 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.320053 23305 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.314960 23232 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:04.313867 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.142503 23296 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.147096 23269 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:59:04.403513 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:04.341162 20496 raft_consensus.cc:1273] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:59:04.352952 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.382486 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:04.341676 20501 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20250831 13:59:04.460247 20568 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:04.383400 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:04.408613 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 100006us
I20250831 13:59:04.322118 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.438987 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1568) took 2068 ms. Trace:
I20250831 13:59:04.463011 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:04.205205 20541 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.131:45067 timed out after 1.659s (SENT) [suppressed 25 similar messages]
I20250831 13:59:04.464540 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.471138 20569 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:04.465550 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:04.463399 20574 rpcz_store.cc:276] 0831 13:59:02.370576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.370998 (+ 422us) service_pool.cc:225] Handling call
0831 13:59:02.420302 (+ 49304us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:02.500187 (+ 79885us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:02.500216 (+ 29us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:02.500699 (+ 483us) log.cc:844] Serialized 26 byte log entry
0831 13:59:03.154778 (+654079us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:59:03.154818 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:03.154918 (+ 100us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:04.376514 (+1221596us) raft_consensus.cc:1683] finished
0831 13:59:04.376541 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:04.438928 (+ 62387us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":192415,"raft.queue_time_us":92109,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":112531,"thread_start_us":349261,"threads_started":1,"wal-append.queue_time_us":1288859}
I20250831 13:59:04.468796 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:59:04.194434 23003 consensus_meta.cc:287] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.683s user 0.004s sys 0.006s
I20250831 13:59:04.458391 20573 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:59:04.462930 20645 raft_consensus.cc:2466] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:04.324031 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.462210 20649 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
W20250831 13:59:04.340950 23264 consensus_meta.cc:287] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.830s user 0.010s sys 0.001s
W20250831 13:59:04.185516 23228 log.cc:927] Time spent T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 2.136s user 0.004s sys 0.019s
I20250831 13:59:04.518044 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:04.518885 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 82065.5us
I20250831 13:59:04.519084 20573 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:04.333647 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.519728 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1511) took 1004 ms. Trace:
W20250831 13:59:04.443322 23302 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.262s user 0.000s sys 0.000s
I20250831 13:59:04.520232 20569 rpcz_store.cc:276] 0831 13:59:03.515260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.519000 (+1003740us) service_pool.cc:225] Handling call
0831 13:59:04.519279 (+ 279us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:04.519441 (+ 162us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:04.519464 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:04.519512 (+ 48us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:04.519544 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:04.519593 (+ 49us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:04.519682 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:04.520090 20494 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
W20250831 13:59:04.519937 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 48229.7us
I20250831 13:59:04.190457 23173 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 12 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:04.254828 23213 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.195662 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:04.469836 22859 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.302893 23167 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.171438 23000 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.523949 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:04.377601 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:04.190940 23224 raft_consensus.cc:3053] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:04.529384 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:04.541465 23184 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.396s user 0.010s sys 0.026s
W20250831 13:59:04.543620 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 378403us
I20250831 13:59:04.549152 20645 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250831 13:59:04.549680 20617 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:04.374434 23125 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.223282 23268 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:04.556964 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.373641 23304 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.528864 20501 raft_consensus.cc:1273] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:59:04.585278 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.433648 23287 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.438963 22371 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:04.559280 20568 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:04.587229 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.438859 23311 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.321118 23257 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.402411 22804 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.184569 23282 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.179690 22900 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:04.382721 23239 log.cc:927] Time spent T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 2.334s user 0.004s sys 0.025s
I20250831 13:59:04.448179 23154 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:04.371326 23310 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:04.374825 23258 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:59:04.566561 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.568588 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1509) took 1053 ms. Trace:
W20250831 13:59:04.567497 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 362314us
I20250831 13:59:04.379977 23200 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:04.571643 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 47807.8us
I20250831 13:59:04.444173 23232 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:04.147830 23289 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:59:04.473240 23209 consensus_meta.cc:287] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 1.680s user 0.005s sys 0.006s
I20250831 13:59:04.573637 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:04.166083 23106 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.575085 20574 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:04.611200 20645 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:59:04.378831 23225 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.261350 23217 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:04.154268 23284 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.322342 23141 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.184404 23297 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:04.573762 20494 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:04.624768 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.322122 23058 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:04.572836 23184 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.430s user 0.011s sys 0.026s
I20250831 13:59:04.592856 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:04.600101 22900 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:04.594563 20568 raft_consensus.cc:1273] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:04.595685 20647 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 8.
I20250831 13:59:04.551544 23305 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.597945 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.576067 22554 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:04.572731 23224 raft_consensus.cc:738] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:04.326085 23245 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.572918 23167 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:04.526436 23264 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:59:04.572692 21208 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:04.602583 20575 rpcz_store.cc:276] 0831 13:59:03.514720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.094945 (+580225us) service_pool.cc:225] Handling call
0831 13:59:04.111864 (+ 16919us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:04.112158 (+ 294us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:59:04.480369 (+368211us) log.cc:844] Serialized 14 byte log entry
0831 13:59:04.568289 (+ 87920us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:04.568337 (+ 48us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:04.568379 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:04.568435 (+ 56us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:04.568535 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":79377,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":1803,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":16014,"thread_start_us":267543,"threads_started":2}
I20250831 13:59:04.199016 22613 consensus_queue.cc:237] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.607100 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.187757 23166 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.612871 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:04.612766 23209 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:04.595180 23311 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.595486 23257 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.472805 23296 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.563526 23213 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.606079 21174 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:04.600134 23312 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.615170 23106 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.615697 23321 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:04.615697 20571 raft_consensus.cc:1238] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:04.616822 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:04.602890 23154 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.472918 23292 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.566850 22859 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.573108 23000 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.628108 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.594381 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.593838 22804 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.602061 23282 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:04.589572 23304 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:04.627247 23173 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.836256 23173 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.837733 23296 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.820868 23284 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.639263 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.839109 20649 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:04.839344 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.815224 23288 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:04.715219 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 86389.1us
I20250831 13:59:04.856841 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.770651 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.668432 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1541) took 3554 ms. Trace:
I20250831 13:59:04.781869 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1603) took 2429 ms. Trace:
W20250831 13:59:04.599702 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 155600us
I20250831 13:59:04.842559 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.843675 20494 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
W20250831 13:59:04.843729 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:04.601521 21206 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47746 (request call id 93) took 2252 ms (client timeout 2197 ms). Trace:
I20250831 13:59:04.639771 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:04.844522 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 188653us
I20250831 13:59:04.876611 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:04.720887 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1551) took 3420 ms. Trace:
I20250831 13:59:04.877149 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.678445 20645 raft_consensus.cc:2408] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250831 13:59:04.772841 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.857702 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.878619 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.878823 20651 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:04.878789 20466 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:04.858943 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:04.862183 20568 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:04.879673 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:04.879976 20647 raft_consensus.cc:2391] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 12: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:04.858073 20570 rpcz_store.cc:276] 0831 13:59:01.113632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.160165 (+ 46533us) service_pool.cc:225] Handling call
0831 13:59:01.201463 (+ 41298us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:01.201880 (+ 417us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:01.201923 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:01.631156 (+429233us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b2c00000c88. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bb3199a1 00007fb9bb328fd2 00007fb9bc9b2021 00007fb9bc9763fa 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32
0831 13:59:01.660752 (+ 29596us) log.cc:844] Serialized 26 byte log entry
0831 13:59:01.926165 (+265413us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:01.937901 (+ 11736us) log.cc:844] Serialized 14 byte log entry
0831 13:59:01.940926 (+ 3025us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:01.941020 (+ 94us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:02.855024 (+914004us) spinlock_profiling.cc:243] Waited 9.48 ms on lock 0x7b18002598a0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b9a14792 00007fb9b9a16b61 00007fb9b9a16b19 00007fb9b9a16aa9 00007fb9b9a16a71 00007fb9b9a15d6d 0000000000379254 0000000000379099 00007fb9b99f7b1a
0831 13:59:04.184940 (+1329916us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:04.668191 (+483251us) raft_consensus.cc:1683] finished
0831 13:59:04.668218 (+ 27us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:04.668370 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":369068,"prepare.queue_time_us":17918,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":69452,"raft.run_cpu_time_us":300,"raft.run_wall_time_us":269683,"spinlock_wait_cycles":299826048,"thread_start_us":209121,"threads_started":1,"wal-append.queue_time_us":345175}
I20250831 13:59:04.879235 20642 raft_consensus.cc:2408] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20250831 13:59:04.639039 21209 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:04.856482 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.880574 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:04.881191 20643 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:04.880460 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:04.858222 20493 rpcz_store.cc:276] 0831 13:59:02.352280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.352488 (+ 208us) service_pool.cc:225] Handling call
0831 13:59:02.352735 (+ 247us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:03.448479 (+1095744us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:04.781517 (+1333038us) spinlock_profiling.cc:243] Waited 518 ms on lock 0x7b540035acd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:04.781751 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1192052992}
W20250831 13:59:04.774468 23254 log.cc:927] Time spent T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 2.107s user 0.002s sys 0.022s
I20250831 13:59:04.862641 23158 raft_consensus.cc:3053] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:04.606302 23310 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:04.885039 20650 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 1095) took 2895 ms (client timeout 2648 ms). Trace:
I20250831 13:59:04.882325 20577 raft_consensus.cc:1238] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20250831 13:59:04.882170 20568 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:04.882467 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:04.886394 20647 raft_consensus.cc:1273] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20250831 13:59:04.883168 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.618671 23306 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.697139 23281 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.666602 22819 consensus_queue.cc:1046] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.237s
I20250831 13:59:04.884613 23166 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:04.780485 22815 raft_consensus.cc:3053] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:04.877467 20576 rpcz_store.cc:276] 0831 13:59:01.301049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.377527 (+ 76478us) service_pool.cc:225] Handling call
0831 13:59:01.407198 (+ 29671us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:01.808003 (+400805us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:01.808048 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:01.845344 (+ 37296us) log.cc:844] Serialized 26 byte log entry
0831 13:59:01.996091 (+150747us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:02.205552 (+209461us) log.cc:844] Serialized 14 byte log entry
0831 13:59:02.243221 (+ 37669us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:02.243290 (+ 69us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:04.381946 (+2138656us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:04.716994 (+335048us) raft_consensus.cc:1683] finished
0831 13:59:04.720680 (+ 3686us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:04.720819 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44067,"prepare.queue_time_us":104742,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":247015,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":51296,"spinlock_wait_cycles":245793536,"thread_start_us":635097,"threads_started":3,"wal-append.queue_time_us":178208}
I20250831 13:59:04.917889 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:04.918459 20576 raft_consensus.cc:3058] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Advancing to term 12
I20250831 13:59:04.918838 23292 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:04.921818 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 64191.8us
I20250831 13:59:04.721881 23269 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:04.907160 20495 raft_consensus.cc:2391] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:04.904572 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.927080 20645 raft_consensus.cc:2391] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:59:04.927434 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 48762.5us
I20250831 13:59:04.927860 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.928627 20645 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 5.
I20250831 13:59:04.883996 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.904258 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:04.885546 20650 rpcz_store.cc:269] 0831 13:59:01.989909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.006557 (+ 16648us) service_pool.cc:225] Handling call
0831 13:59:04.770487 (+2763930us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b5800105c98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:59:04.884971 (+114484us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28292992}
I20250831 13:59:04.910948 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.930810 20574 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:04.909363 20493 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
W20250831 13:59:04.931437 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 54916.8us
I20250831 13:59:04.880460 20494 raft_consensus.cc:2391] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 12: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:04.913055 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.932545 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:04.876019 21206 rpcz_store.cc:269] 0831 13:59:02.349230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.355607 (+ 6377us) service_pool.cc:225] Handling call
0831 13:59:04.377427 (+2021820us) spinlock_profiling.cc:243] Waited 5.74 ms on lock 0x7b58000c0098. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:59:04.601440 (+224013us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13195136}
I20250831 13:59:04.917037 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1593) took 2894 ms. Trace:
I20250831 13:59:04.861011 23321 raft_consensus.cc:695] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:04.913057 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:04.922556 20616 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:04.641561 23003 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.924671 20576 raft_consensus.cc:2466] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 12.
I20250831 13:59:04.989573 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.905162 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.990986 20646 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
W20250831 13:59:04.862497 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 267570us
I20250831 13:59:04.844767 22554 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.004532 22554 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:04.856361 23167 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.886348 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.006992 20575 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:05.007659 23167 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.928809 20466 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:05.007818 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:04.862182 23312 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.008494 20575 raft_consensus.cc:2391] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:05.009260 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.009740 23312 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.009811 20575 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:04.933465 20568 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:05.010772 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.734223 23287 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:04.934715 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.934406 20650 raft_consensus.cc:1273] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250831 13:59:04.944336 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:04.969233 20500 rpcz_store.cc:276] 0831 13:59:02.028851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.029009 (+ 158us) service_pool.cc:225] Handling call
0831 13:59:02.029341 (+ 332us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:02.114945 (+ 85604us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:04.916791 (+2801846us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b54001c3254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:04.916979 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":134055680}
I20250831 13:59:05.016866 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:04.933812 22815 raft_consensus.cc:738] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:04.679052 23232 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:04.881609 22873 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.033074 22873 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.033571 22873 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.034310 23282 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:04.987353 23289 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.984120 20643 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20250831 13:59:04.984755 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 52632.7us
I20250831 13:59:05.041592 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.860772 21203 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
W20250831 13:59:04.991668 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 69193.3us
I20250831 13:59:05.011191 22371 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:04.992638 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:04.993466 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:04.994192 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:04.994740 23154 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.046800 20616 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:04.995448 23310 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.995848 22859 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.995975 21205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47742 (request call id 114) took 2974 ms. Trace:
I20250831 13:59:04.997027 23245 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.998514 23213 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:04.884207 23158 raft_consensus.cc:738] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:05.055863 23158 consensus_queue.cc:260] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.057613 23158 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:05.057961 23141 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.040884 23230 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:05.045320 23259 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:05.059521 23259 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.060102 23259 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:05.058705 21205 rpcz_store.cc:276] 0831 13:59:02.021202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.032494 (+ 11292us) service_pool.cc:225] Handling call
0831 13:59:02.041366 (+ 8872us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:02.063398 (+ 22032us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:02.079816 (+ 16418us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:02.102265 (+ 22449us) log.cc:844] Serialized 26 byte log entry
0831 13:59:02.247430 (+145165us) raft_consensus.cc:1643] Marking committed up to 8
0831 13:59:02.259801 (+ 12371us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:02.259924 (+ 123us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:04.773896 (+2513972us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:04.995644 (+221748us) raft_consensus.cc:1683] finished
0831 13:59:04.995688 (+ 44us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:04.995898 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11459,"raft.queue_time_us":18706,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":24414,"spinlock_wait_cycles":6120832,"thread_start_us":68288,"threads_started":1,"wal-append.queue_time_us":187392}
W20250831 13:59:05.045163 23326 log.cc:927] Time spent T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20250831 13:59:05.064782 21174 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:59:05.065404 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.065594 23273 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:05.066114 23273 raft_consensus.cc:695] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 8 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:59:04.843528 23058 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.002364 20651 raft_consensus.cc:1273] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:59:05.011327 23269 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20250831 13:59:05.002211 23003 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.664819 23297 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.986860 23311 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.012514 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.077697 20575 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
W20250831 13:59:05.067027 20602 heartbeater.cc:646] Failed to heartbeat to 127.18.233.190:42663 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.18.233.190:42663 timed out after 15.000s (SENT)
I20250831 13:59:04.912173 23173 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.011318 20568 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:04.609761 23258 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:05.018448 20500 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
W20250831 13:59:05.018914 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 90210.2us
I20250831 13:59:04.860489 23316 consensus_queue.cc:1046] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.143s
I20250831 13:59:04.979964 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.109516 20494 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:05.109568 20466 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:05.020706 20501 raft_consensus.cc:1273] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:04.941879 23106 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.987614 23127 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20250831 13:59:05.026198 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:04.935995 23204 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:04.988165 23217 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.030090 23305 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.029668 23321 consensus_queue.cc:237] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.042089 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.099732 23311 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.042773 20617 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
W20250831 13:59:05.124536 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 81841.7us
I20250831 13:59:05.126013 20617 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:05.041067 23227 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 2.960s user 0.025s sys 0.027s
I20250831 13:59:05.045085 23216 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 13:59:05.045292 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 52780.5us
W20250831 13:59:05.045285 23327 log.cc:927] Time spent T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250831 13:59:05.045897 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:05.045183 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.044732 20576 raft_consensus.cc:2391] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 12: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:05.047956 20574 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:59:05.051612 20616 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:05.108587 23141 raft_consensus.cc:491] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.112944 23284 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:04.911929 23288 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.071878 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:04.861773 21208 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47746 (request call id 92) took 2534 ms (client timeout 2092 ms). Trace:
I20250831 13:59:05.157398 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.046747 22371 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:04.989447 23224 consensus_queue.cc:260] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:04.925614 23306 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.095366 22554 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.095355 22804 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.099124 20499 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:59:05.071610 23268 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:04.978874 23281 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.042903 23232 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.112875 20466 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:05.123487 20501 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250831 13:59:04.880630 22900 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.027217 22882 raft_consensus.cc:2747] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250831 13:59:05.132869 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 81393.8us
I20250831 13:59:05.131497 20648 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:59:05.137048 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.216114 20616 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.217082 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.217634 20616 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:05.139804 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.219987 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.141930 23304 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.142185 23182 raft_consensus.cc:3053] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Stepping down as leader of term 2
I20250831 13:59:05.142941 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1549) took 3886 ms. Trace:
I20250831 13:59:05.123407 23264 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.123278 23000 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.143000 21203 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:05.147481 23173 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.116839 23106 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:05.113636 23127 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.115386 23167 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:05.069057 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.254626 20643 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:05.152405 23246 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250831 13:59:05.136157 23316 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:05.151520 23230 consensus_queue.cc:1046] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20250831 13:59:05.148952 23003 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.167917 22873 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:59:05.169654 23317 log.cc:927] Time spent T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.575s user 0.001s sys 0.000s
I20250831 13:59:05.169869 23204 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.114615 23077 consensus_queue.cc:1035] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20250831 13:59:05.171454 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:05.173539 23151 raft_consensus.cc:3058] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:05.174104 20541 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:05.255280 23127 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:05.175163 23231 consensus_queue.cc:1046] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:05.175770 23257 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.293797 22901 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.130411 23305 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.295773 20569 raft_consensus.cc:1273] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20250831 13:59:05.296306 23305 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.141937 23258 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:05.297385 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1575) took 1145 ms. Trace:
I20250831 13:59:05.297736 20569 rpcz_store.cc:276] 0831 13:59:04.152330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.529380 (+377050us) service_pool.cc:225] Handling call
0831 13:59:04.529625 (+ 245us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.296162 (+766537us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.297173 (+ 1011us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b54002a8fd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:05.297332 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1761257216}
I20250831 13:59:05.132454 23311 raft_consensus.cc:513] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.299707 23268 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.300144 23268 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.300360 23311 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.151680 23141 raft_consensus.cc:513] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.302253 23141 leader_election.cc:290] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.139647 23297 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.180724 23284 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.303136 23284 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.303573 20568 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20250831 13:59:05.304792 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.305183 23284 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.181916 22815 consensus_queue.cc:260] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.305455 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:05.305915 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:05.185209 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.128396 23209 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:59:05.205922 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.206044 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:05.212936 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 100116us
I20250831 13:59:05.222122 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.311553 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.311767 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.205560 23296 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.185968 23269 consensus_queue.cc:1046] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250831 13:59:05.205492 23213 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.229178 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.223562 20572 rpcz_store.cc:276] 0831 13:59:01.256997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.334247 (+ 77250us) service_pool.cc:225] Handling call
0831 13:59:01.382981 (+ 48734us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:01.611686 (+228705us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:01.612565 (+ 879us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:01.630339 (+ 17774us) log.cc:844] Serialized 26 byte log entry
0831 13:59:01.935555 (+305216us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:02.105320 (+169765us) log.cc:844] Serialized 14 byte log entry
0831 13:59:02.121535 (+ 16215us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:02.121615 (+ 80us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:05.004181 (+2882566us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:05.142572 (+138391us) raft_consensus.cc:1683] finished
0831 13:59:05.142695 (+ 123us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.142865 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":89438,"prepare.queue_time_us":128433,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":235967,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":29969,"spinlock_wait_cycles":277815040,"thread_start_us":586880,"threads_started":3,"wal-append.queue_time_us":367502}
I20250831 13:59:05.228552 20642 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:05.067004 23273 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:05.207069 22780 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.216768 23158 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:05.252060 20616 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:59:05.329527 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 77581.3us
I20250831 13:59:05.330305 20616 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.331104 20616 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.188148 23013 consensus_queue.cc:1046] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20250831 13:59:05.336117 20497 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:05.337090 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.273089 23246 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.338557 20616 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.340106 20616 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.255484 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.186694 23288 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.276603 20576 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 5.
I20250831 13:59:05.248039 21207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:05.249037 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:59:05.279498 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 105455us
I20250831 13:59:05.311442 21206 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
W20250831 13:59:05.280398 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 109007us
I20250831 13:59:05.223310 23182 raft_consensus.cc:738] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:05.231124 23315 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.230945 23173 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:05.330443 22554 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.254000 21174 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:05.286641 23306 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.287891 23106 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:05.283402 23204 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:05.280045 23003 raft_consensus.cc:3053] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:05.370744 23288 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.370886 23003 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:05.372044 22873 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.283630 23230 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.372826 23230 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:59:05.373253 23230 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.373790 23204 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:05.375192 22369 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.375640 22850 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.375787 22369 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.376174 22850 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.283218 23197 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.380551 23197 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.380988 23197 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.231274 22804 consensus_queue.cc:1035] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20250831 13:59:05.359445 21176 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.306223 22815 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:05.178557 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34436 (request call id 74) took 2900 ms. Trace:
I20250831 13:59:05.390722 20577 rpcz_store.cc:276] 0831 13:59:02.277910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.313039 (+ 35129us) service_pool.cc:225] Handling call
0831 13:59:02.373134 (+ 60095us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:04.885928 (+2512794us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.178282 (+292354us) spinlock_profiling.cc:243] Waited 2.51 s on lock 0x7b54001fac54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:05.178489 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5770102656}
I20250831 13:59:05.307369 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.270251 23316 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.398922 23316 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.399773 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:05.310222 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.400826 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:05.401285 20543 connection.cc:232] RPC call timeout handler was delayed by -1.8452933869999697s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2025s for a response
I20250831 13:59:05.401803 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:05.275743 23000 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:05.403074 23000 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.405020 23000 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.184296 22371 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:05.316972 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.409806 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.318871 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:05.158069 21208 rpcz_store.cc:269] 0831 13:59:02.324383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:02.325116 (+ 733us) service_pool.cc:225] Handling call
0831 13:59:04.195509 (+1870393us) spinlock_profiling.cc:243] Waited 4.39 ms on lock 0x7b58000c0098. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d75ce 000000000035a3c9 00007fb9c185e300 00007fb9c1924ec4 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:59:04.844472 (+648963us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10106624}
I20250831 13:59:05.411309 23306 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.411633 22979 raft_consensus.cc:3058] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.411785 23306 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.413203 22999 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.413812 23306 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.413754 22999 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.415647 22999 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.416430 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.363763 23173 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.343241 20648 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [term 9 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:05.345463 20572 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0"
dest_uuid: "212b66976f874aa682cae0fb77928786"
mode: GRACEFUL
new_leader_uuid: "58aded50ab0d4609a053647eb11b1f48"
from {username='slave'} at 127.0.0.1:49676
I20250831 13:59:05.348582 20541 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:05.418659 20543 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:05.328846 23258 consensus_queue.cc:1046] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20250831 13:59:05.355048 20500 raft_consensus.cc:1273] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:05.329150 23158 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.336941 23269 consensus_queue.cc:1230] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [NON_LEADER]: Peer 58aded50ab0d4609a053647eb11b1f48 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250831 13:59:05.318522 23111 consensus_queue.cc:1035] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20250831 13:59:05.343937 23287 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:05.310381 23259 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.344609 23246 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:59:05.359198 23266 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.009s
I20250831 13:59:05.346498 23321 raft_consensus.cc:2802] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 12 FOLLOWER]: Leader election won for term 12
I20250831 13:59:05.348621 23216 raft_consensus.cc:2802] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:59:05.348583 23219 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.371940 23003 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.428689 23158 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.379805 22850 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.353705 23270 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:05.355324 23315 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.390115 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.433193 23281 leader_election.cc:290] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.337105 23213 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.395545 23289 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.336256 22780 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.475680 22900 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.398298 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.474483 22369 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.503093 20645 raft_consensus.cc:2408] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20250831 13:59:05.488865 23244 log.cc:927] Time spent T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602: Append to log took a long time: real 3.140s user 0.011s sys 0.013s
I20250831 13:59:05.341892 20643 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:05.504144 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.485390 23219 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.419214 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:05.388972 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 161107us
W20250831 13:59:05.421326 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 72828.9us
I20250831 13:59:05.509797 20541 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:05.122591 23217 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.423207 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:05.423209 20543 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:05.378954 22554 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.428867 21209 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47742 (request call id 115) took 1306 ms. Trace:
I20250831 13:59:05.431375 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.432687 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:59:05.460903 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 101003us
I20250831 13:59:05.419610 22873 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:05.462862 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.520462 20649 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:05.521253 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.392146 23284 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.422346 23204 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.427733 22999 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250831 13:59:05.322391 23296 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.427069 23288 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:05.428025 23312 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.429351 23229 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:05.435737 23287 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.491564 20502 raft_consensus.cc:1238] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:05.494926 20493 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:59:05.547055 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.391775 20569 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:05.432952 23232 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.471836 23245 raft_consensus.cc:3053] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:05.396111 23182 consensus_queue.cc:260] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.496280 23315 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:59:05.464500 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 117350us
I20250831 13:59:05.560407 20574 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:05.505229 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.561764 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:05.343842 23077 consensus_queue.cc:1035] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20250831 13:59:05.486497 23321 raft_consensus.cc:695] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [term 12 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:05.471568 23268 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:05.512657 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 89515.9us
I20250831 13:59:05.492437 23216 raft_consensus.cc:695] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:05.516744 20568 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 11.
I20250831 13:59:05.518627 20541 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:05.515507 21209 rpcz_store.cc:276] 0831 13:59:04.122418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.122602 (+ 184us) service_pool.cc:225] Handling call
0831 13:59:05.428674 (+1306072us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b540040e054. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:05.428773 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":777868544}
I20250831 13:59:05.519572 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.566541 20543 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.502065 23289 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.505626 23003 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:05.515717 23306 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:05.529636 23284 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.530310 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.525097 22554 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.531026 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.534895 20570 raft_consensus.cc:2408] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20250831 13:59:05.548645 23312 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.558646 23245 raft_consensus.cc:738] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:05.539295 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:05.577741 23312 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.579521 23268 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.581007 23245 consensus_queue.cc:260] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.582084 23245 raft_consensus.cc:3058] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:05.583900 23284 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.584774 23164 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.585561 22554 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.591799 23296 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.569347 23289 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.574442 23141 consensus_queue.cc:1035] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.047s
I20250831 13:59:05.599399 23216 consensus_queue.cc:237] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.571015 22999 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.623019 22999 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.622807 23158 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.623459 22999 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.624722 23158 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.625053 20643 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:05.593375 23259 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.627233 20496 raft_consensus.cc:1238] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
W20250831 13:59:05.572628 23347 log.cc:927] Time spent T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
W20250831 13:59:05.540156 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.540741 23213 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.627863 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 1034) took 1191 ms. Trace:
I20250831 13:59:05.571478 23306 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.628265 20496 rpcz_store.cc:276] 0831 13:59:04.436356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.459219 (+ 22863us) service_pool.cc:225] Handling call
0831 13:59:05.626887 (+1167668us) spinlock_profiling.cc:243] Waited 7.58 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:59:05.627064 (+ 177us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.627637 (+ 573us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.627803 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17426688}
I20250831 13:59:05.628844 23306 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:05.595567 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 1029) took 1268 ms. Trace:
I20250831 13:59:05.597250 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1574) took 1445 ms. Trace:
I20250831 13:59:05.630475 20497 rpcz_store.cc:276] 0831 13:59:04.327668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.327838 (+ 170us) service_pool.cc:225] Handling call
0831 13:59:04.328288 (+ 450us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.336642 (+1008354us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.595335 (+258693us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54000997d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:05.595489 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2317399936}
I20250831 13:59:05.511302 23219 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.493705 22371 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.636153 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.636663 23259 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:05.637238 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.637553 22371 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:05.520126 23338 log.cc:927] Time spent T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20250831 13:59:05.544860 20651 raft_consensus.cc:1238] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: []
W20250831 13:59:05.548979 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.571815 23231 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:05.573496 23159 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.639533 20642 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:05.639816 23159 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.549957 20572 raft_consensus.cc:604] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Received request to transfer leadership to TS 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.555002 20649 raft_consensus.cc:1273] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20250831 13:59:05.558775 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1426) took 3580 ms. Trace:
I20250831 13:59:05.593065 23217 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.643270 23321 consensus_queue.cc:237] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.644198 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1613) took 1209 ms. Trace:
I20250831 13:59:05.643179 20644 rpcz_store.cc:276] 0831 13:59:01.971903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:01.988618 (+ 16715us) service_pool.cc:225] Handling call
0831 13:59:01.988899 (+ 281us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:01.989214 (+ 315us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:01.989242 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:02.016801 (+ 27559us) log.cc:844] Serialized 26 byte log entry
0831 13:59:02.106593 (+ 89792us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:02.106700 (+ 107us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:02.106837 (+ 137us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:03.367563 (+1260726us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b4800061ed8. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c1838c41 00007fb9c182fde7 00007fb9c182fc7a 00007fb9c192506f 00007fb9c1924d37 00007fb9c193935c 00007fb9c1939308 00007fb9c1939288 00007fb9c1939243 00007fb9c19384ec 00007fb9c08b6656
0831 13:59:05.402475 (+2034912us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:05.551710 (+149235us) raft_consensus.cc:1683] finished
0831 13:59:05.551732 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.551873 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":88471,"raft.queue_time_us":4153,"raft.run_cpu_time_us":4382,"raft.run_wall_time_us":1369013,"spinlock_wait_cycles":85701248,"thread_start_us":64884,"threads_started":1,"wal-append.queue_time_us":224530}
W20250831 13:59:05.560854 20540 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.563246 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.646512 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.647524 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:05.647662 23259 consensus_queue.cc:1046] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:05.648707 23259 raft_consensus.cc:3053] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:05.649048 23259 raft_consensus.cc:738] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:05.544381 23288 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.572930 22979 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20250831 13:59:05.649549 23288 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.649734 20493 raft_consensus.cc:1238] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:05.542599 23204 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.547816 23287 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:05.652132 23204 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:05.568984 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 50412.6us
I20250831 13:59:05.653966 20541 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.601027 23224 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.569975 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.656040 20541 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:05.656080 23315 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:05.575736 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.577046 22780 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.684616 21175 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:05.618700 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1030) took 1286 ms. Trace:
I20250831 13:59:05.686277 20499 rpcz_store.cc:276] 0831 13:59:04.332102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.332221 (+ 119us) service_pool.cc:225] Handling call
0831 13:59:05.065243 (+733022us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18d3b50 00007fb9c18d3434 00007fb9baf620c4 00007fb9baf62052
0831 13:59:05.618405 (+553162us) spinlock_profiling.cc:243] Waited 33.1 ms on lock 0x7b540021c5d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:05.618556 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":130153600}
I20250831 13:59:05.632501 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.687466 20497 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:05.541219 20577 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 13:59:05.688290 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.637946 20496 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:05.644688 20642 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250831 13:59:05.568356 23182 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 2 FOLLOWER]: Advancing to term 3
I20250831 13:59:05.642865 20572 raft_consensus.cc:612] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:59:05.620635 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1621) took 1097 ms. Trace:
I20250831 13:59:05.690382 20494 rpcz_store.cc:276] 0831 13:59:04.523174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.110180 (+587006us) service_pool.cc:225] Handling call
0831 13:59:05.618889 (+508709us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.619108 (+ 219us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.619143 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:05.619199 (+ 56us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:05.619243 (+ 44us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.619326 (+ 83us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.620474 (+ 1148us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b540021c5d0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:05.620579 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1169280384}
I20250831 13:59:05.657269 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.596597 23230 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:05.642905 20649 pending_rounds.cc:85] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Aborting all ops after (but not including) 4
I20250831 13:59:05.657932 22979 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:59:05.535979 22901 raft_consensus.cc:3053] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:05.658849 23231 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.659714 23313 consensus_queue.cc:1035] T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 13:59:05.659934 23127 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.732192 23127 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.661692 20570 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41400 (request call id 1525) took 1038 ms. Trace:
I20250831 13:59:05.661947 20573 raft_consensus.cc:1273] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20250831 13:59:05.662310 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1028) took 1402 ms. Trace:
I20250831 13:59:05.732771 20570 rpcz_store.cc:276] 0831 13:59:04.623155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.016007 (+392852us) service_pool.cc:225] Handling call
0831 13:59:05.661495 (+645488us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54000b45d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:05.661638 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1132439808}
I20250831 13:59:05.733188 20495 rpcz_store.cc:276] 0831 13:59:04.259678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.259809 (+ 131us) service_pool.cc:225] Handling call
0831 13:59:05.662128 (+1402319us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b5400011dd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:05.662254 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1018133888}
I20250831 13:59:05.644562 20502 rpcz_store.cc:276] 0831 13:59:04.434936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.435110 (+ 174us) service_pool.cc:225] Handling call
0831 13:59:05.064093 (+628983us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:59:05.064283 (+ 190us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.547798 (+483515us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.643994 (+ 96196us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b54001284d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:05.644146 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":989969664}
I20250831 13:59:05.663586 23204 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 13:59:05.664114 23336 log.cc:927] Time spent T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.267s user 0.003s sys 0.000s
I20250831 13:59:05.734884 23204 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.735311 23204 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.569320 20543 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:59:05.736245 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 166966us
I20250831 13:59:05.704109 23106 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:05.740391 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:05.590564 23316 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.664538 23173 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:05.665158 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 530780us
I20250831 13:59:05.745088 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.745324 20467 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:59:05.675673 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 45886.5us
I20250831 13:59:05.745729 20574 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:05.680959 20468 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:05.630659 20571 rpcz_store.cc:276] 0831 13:59:04.152026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.528076 (+376050us) service_pool.cc:225] Handling call
0831 13:59:04.528758 (+ 682us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:04.844373 (+315615us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.597004 (+752631us) spinlock_profiling.cc:243] Waited 86.7 ms on lock 0x7b54001fa754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:05.597183 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":199465856}
I20250831 13:59:05.748919 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.749668 20571 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:05.695202 20649 pending_rounds.cc:107] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20250831 13:59:05.750528 20649 raft_consensus.cc:2887] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:05.699371 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1513) took 1080 ms. Trace:
I20250831 13:59:05.688938 20497 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
W20250831 13:59:05.671403 23343 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20250831 13:59:05.761395 23083 consensus_queue.cc:1046] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 13:59:05.714278 23289 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:05.764101 23289 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.764974 23306 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.766111 20569 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
W20250831 13:59:05.684435 23349 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20250831 13:59:05.768291 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1514) took 1149 ms. Trace:
I20250831 13:59:05.704998 23243 consensus_queue.cc:1046] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20250831 13:59:05.768777 20569 rpcz_store.cc:276] 0831 13:59:04.619041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.550183 (+931142us) service_pool.cc:225] Handling call
0831 13:59:05.562763 (+ 12580us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.766473 (+203710us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.768056 (+ 1583us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b540031ecd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:05.768212 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":467184512}
I20250831 13:59:05.769788 21201 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:05.770293 20647 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:05.705257 22901 raft_consensus.cc:738] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:05.772939 22901 consensus_queue.cc:260] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.773520 23106 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.773864 22901 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.774221 23106 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.603340 23197 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.690092 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:05.685448 23342 log.cc:927] Time spent T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.125s user 0.000s sys 0.001s
I20250831 13:59:05.781767 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1584) took 1618 ms. Trace:
I20250831 13:59:05.715198 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.671404 23158 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.715937 23230 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:05.668012 23273 consensus_queue.cc:1046] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20250831 13:59:05.716202 22873 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:05.717049 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.721269 22999 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.726342 20498 raft_consensus.cc:2408] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 13:59:05.728411 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.732873 20573 pending_rounds.cc:85] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 3
I20250831 13:59:05.734473 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.751996 20577 rpcz_store.cc:276] 0831 13:59:04.618825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.527713 (+908888us) service_pool.cc:225] Handling call
0831 13:59:05.540978 (+ 13265us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:05.699070 (+158092us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.699302 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:05.822724 20573 pending_rounds.cc:107] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20250831 13:59:05.822991 20499 raft_consensus.cc:2408] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:59:05.823055 20573 raft_consensus.cc:2887] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:05.743253 22979 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:05.755115 23173 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.755756 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.757280 23287 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:05.742251 23231 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.758580 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.823017 20570 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:05.827037 20574 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:59:05.759289 22780 raft_consensus.cc:491] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.759449 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.760195 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:05.760489 23316 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.666632 23219 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.847371 23219 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:05.765331 23083 raft_consensus.cc:3053] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:05.848003 23083 raft_consensus.cc:738] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:05.849041 23083 consensus_queue.cc:260] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:59:05.712896 23315 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:05.777673 20467 leader_election.cc:304] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:59:05.851769 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 74179.6us
I20250831 13:59:05.852669 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:05.853905 20467 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:05.776149 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.854791 20569 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:59:05.855610 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.856069 23197 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.856642 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.857260 20569 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
W20250831 13:59:05.777187 20468 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 96338.1us
I20250831 13:59:05.782819 20575 rpcz_store.cc:276] 0831 13:59:04.163183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.084597 (+921414us) service_pool.cc:225] Handling call
0831 13:59:05.113864 (+ 29267us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:05.307932 (+194068us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.307976 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:05.317847 (+ 9871us) log.cc:844] Serialized 26 byte log entry
0831 13:59:05.497703 (+179856us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:05.497754 (+ 51us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.777964 (+280210us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54002fdd54. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:05.778036 (+ 72us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:05.778093 (+ 57us) raft_consensus.cc:1683] finished
0831 13:59:05.778113 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.781707 (+ 3594us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1754,"raft.queue_time_us":1962,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":14590,"spinlock_wait_cycles":445249920,"thread_start_us":146304,"threads_started":1,"wal-append.queue_time_us":220627}
I20250831 13:59:05.862242 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.787541 23265 consensus_queue.cc:1035] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
I20250831 13:59:05.862843 20575 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:59:05.863723 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.863994 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:05.864485 20575 raft_consensus.cc:2408] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:59:05.758580 23313 consensus_queue.cc:1035] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20250831 13:59:05.864951 20568 raft_consensus.cc:2391] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 6: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:05.820768 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:05.889979 20644 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.821800 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:59:05.823874 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 82765.3us
I20250831 13:59:05.828142 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.826289 22979 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.892587 22979 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.893704 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:05.839664 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.821226 22873 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:05.843796 23190 raft_consensus.cc:3053] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:05.851840 20648 raft_consensus.cc:1238] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20250831 13:59:05.865276 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.784724 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:05.866318 20496 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:05.783198 20642 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 9.
I20250831 13:59:05.867368 20497 raft_consensus.cc:1273] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:59:05.867237 23246 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.868075 20617 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:05.868336 22780 raft_consensus.cc:513] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.868937 23182 raft_consensus.cc:2747] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250831 13:59:05.779654 23259 consensus_queue.cc:260] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.786863 22987 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:05.869787 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:05.870117 23167 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:05.870323 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:05.911803 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.870533 20645 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:05.870587 23243 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:05.871026 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.829777 23158 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.871894 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.872047 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:05.873099 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:05.874643 23287 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:05.875910 20650 raft_consensus.cc:2391] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:05.827376 23308 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:05.781356 23277 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:59:05.880043 23231 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.896276 23197 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.886425 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1515) took 1267 ms. Trace:
I20250831 13:59:05.828531 23245 raft_consensus.cc:3053] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:05.828763 23304 raft_consensus.cc:3053] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:05.905679 23292 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:05.891603 20574 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:05.894969 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.940685 20643 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:05.911628 22987 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:05.898375 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:05.943333 20497 raft_consensus.cc:2366] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 8. Current term is 9.
I20250831 13:59:05.943362 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.900051 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.901117 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:05.932691 20576 rpcz_store.cc:276] 0831 13:59:04.619218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.597237 (+978019us) service_pool.cc:225] Handling call
0831 13:59:05.615156 (+ 17919us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:59:05.615904 (+ 748us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.615939 (+ 35us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:59:05.617039 (+ 1100us) log.cc:844] Serialized 52 byte log entry
0831 13:59:05.748751 (+131712us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:59:05.749358 (+ 607us) log.cc:844] Serialized 14 byte log entry
0831 13:59:05.777470 (+ 28112us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.777544 (+ 74us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:05.886113 (+108569us) raft_consensus.cc:1683] finished
0831 13:59:05.886161 (+ 48us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.886354 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27700,"prepare.queue_time_us":389,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":27898,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":7204,"thread_start_us":131342,"threads_started":1,"wal-append.queue_time_us":266558}
I20250831 13:59:05.944581 20571 raft_consensus.cc:2466] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:05.945078 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:05.945425 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1520) took 1323 ms. Trace:
I20250831 13:59:05.900255 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1516) took 1278 ms. Trace:
W20250831 13:59:05.945950 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.945586 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:05.945849 20575 rpcz_store.cc:276] 0831 13:59:04.621508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.899469 (+1277961us) service_pool.cc:225] Handling call
0831 13:59:05.899789 (+ 320us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.899960 (+ 171us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.899984 (+ 24us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:05.900022 (+ 38us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:05.900054 (+ 32us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.900105 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.900206 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:05.909929 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:05.945719 20576 rpcz_store.cc:276] 0831 13:59:04.622288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.944852 (+1322564us) service_pool.cc:225] Handling call
0831 13:59:05.945103 (+ 251us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.945220 (+ 117us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.945234 (+ 14us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:05.945260 (+ 26us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:05.945286 (+ 26us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.945316 (+ 30us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.945386 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:05.912623 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:05.947575 20568 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
I20250831 13:59:05.912576 20467 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:05.915175 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.908816 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1519) took 1286 ms. Trace:
I20250831 13:59:05.918503 20569 raft_consensus.cc:2391] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:05.897147 22873 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:05.921432 20500 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 3.
W20250831 13:59:05.922533 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.951756 20569 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:05.951757 20467 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:59:05.953114 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:05.927155 20646 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20250831 13:59:05.897588 23190 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:05.882860 22999 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.928380 23277 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:05.935046 23308 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:05.962977 20500 raft_consensus.cc:1238] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:05.936746 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.891252 20501 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.941502 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.967331 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:05.968225 20643 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:05.969080 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.850456 23083 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.940253 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:05.919497 22859 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:05.903095 21209 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate ab7ade51fa134a1b814fab748f01d43d in this term.
I20250831 13:59:05.984350 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:05.946077 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:05.947808 20570 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.917569 20617 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:05.985167 20574 raft_consensus.cc:2391] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:59:05.985574 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 68103.6us
I20250831 13:59:05.987692 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:05.988265 20617 leader_election.cc:400] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 8. Current term is 9.
I20250831 13:59:05.988653 20617 leader_election.cc:403] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20250831 13:59:05.908962 20644 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:05.990018 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:05.990505 20568 raft_consensus.cc:1273] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20250831 13:59:05.948743 20576 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:05.948333 20572 rpcz_store.cc:276] 0831 13:59:04.622132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.906598 (+1284466us) service_pool.cc:225] Handling call
0831 13:59:05.907723 (+ 1125us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.908388 (+ 665us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.908429 (+ 41us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:05.908491 (+ 62us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:59:05.908538 (+ 47us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.908614 (+ 76us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.908747 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:05.920742 23173 consensus_queue.cc:1046] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20250831 13:59:05.991914 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1526) took 1368 ms. Trace:
I20250831 13:59:05.941365 23197 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.003137 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:06.003360 20466 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:05.939138 23245 raft_consensus.cc:738] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:05.939302 23304 raft_consensus.cc:738] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:59:06.005287 20571 raft_consensus.cc:1238] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20250831 13:59:05.929566 23306 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:05.972898 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:06.013962 23216 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:05.975250 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.022240 23216 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for earlier term 8. Current term is 9.
I20250831 13:59:05.946619 20495 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:05.991261 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1595) took 1362 ms. Trace:
I20250831 13:59:05.992317 20576 rpcz_store.cc:276] 0831 13:59:04.623984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.947722 (+1323738us) service_pool.cc:225] Handling call
0831 13:59:05.948541 (+ 819us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.991598 (+ 43057us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.991851 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1236224}
I20250831 13:59:06.025125 20568 rpcz_store.cc:276] 0831 13:59:04.628348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.948174 (+1319826us) service_pool.cc:225] Handling call
0831 13:59:05.990111 (+ 41937us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.990983 (+ 872us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.991195 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:05.992769 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:05.994349 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1524) took 1371 ms. Trace:
I20250831 13:59:06.026582 20575 rpcz_store.cc:276] 0831 13:59:04.622990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.947300 (+1324310us) service_pool.cc:225] Handling call
0831 13:59:05.993759 (+ 46459us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.993970 (+ 211us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.993993 (+ 23us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:05.994032 (+ 39us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:05.994070 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.994148 (+ 78us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:05.994287 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:59:06.026867 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 54067.7us
I20250831 13:59:05.954977 23243 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:05.996114 20500 raft_consensus.cc:1273] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20250831 13:59:05.998989 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.002851 20644 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "212b66976f874aa682cae0fb77928786"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:59:06.028569 20644 raft_consensus.cc:604] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Received request to transfer leadership to TS 212b66976f874aa682cae0fb77928786
I20250831 13:59:05.929402 23259 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:05.958606 23277 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:06.003635 20572 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:06.005882 20571 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1523) took 1383 ms. Trace:
I20250831 13:59:05.958789 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1528) took 1242 ms. Trace:
I20250831 13:59:06.113829 20572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1529) took 1324 ms. Trace:
I20250831 13:59:06.113978 20569 rpcz_store.cc:276] 0831 13:59:04.716000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.948719 (+1232719us) service_pool.cc:225] Handling call
0831 13:59:05.951529 (+ 2810us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.958456 (+ 6927us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.958704 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:05.951037 22987 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:05.941180 23292 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:06.118956 23292 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:06.033340 23190 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.119607 23190 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:06.011831 20617 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:06.028213 20500 pending_rounds.cc:85] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 3
I20250831 13:59:06.123062 20500 pending_rounds.cc:107] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20250831 13:59:06.123353 20500 raft_consensus.cc:2887] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:06.123476 23158 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.028046 23257 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:06.124413 23257 raft_consensus.cc:695] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:06.125234 23257 consensus_queue.cc:237] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.127915 23204 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.113675 20571 rpcz_store.cc:276] 0831 13:59:04.622843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.945190 (+1322347us) service_pool.cc:225] Handling call
0831 13:59:05.992281 (+ 47091us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:06.005658 (+ 13377us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.005834 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:06.022841 23316 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:06.029309 20643 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:59:06.031857 20501 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:06.081717 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.080889 23184 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:06.085502 23210 raft_consensus.cc:3053] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:06.143114 23210 raft_consensus.cc:738] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:06.143757 23246 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:06.143965 23210 consensus_queue.cc:260] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.145144 23210 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:59:06.143146 23337 log.cc:927] Time spent T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.731s user 0.000s sys 0.002s
W20250831 13:59:06.147807 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 136040us
I20250831 13:59:06.149152 23083 raft_consensus.cc:2802] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250831 13:59:06.149641 23083 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:06.150046 23083 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:59:06.151504 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:06.151957 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.088558 22873 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:06.152572 20645 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:06.153126 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.153676 20643 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:06.153885 22873 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:06.011201 23315 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:06.154716 20466 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:06.154946 21206 raft_consensus.cc:1238] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:06.155473 23158 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:06.154738 23315 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 212b66976f874aa682cae0fb77928786)
I20250831 13:59:06.155136 22979 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.156368 23315 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
W20250831 13:59:06.157223 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:06.157972 22979 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.158182 23083 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.099403 23304 consensus_queue.cc:260] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:06.159427 20617 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:06.099761 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:59:06.160507 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 60825.4us
I20250831 13:59:06.100451 23245 consensus_queue.cc:260] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.101691 23308 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.094448 22780 leader_election.cc:290] T b2d4d2e85a054797ae54ac949008c1f1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.105535 23288 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.003012 22819 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:06.107355 23197 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.107784 23279 consensus_queue.cc:1046] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250831 13:59:06.108608 23167 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250831 13:59:05.981935 23313 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:06.112047 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:06.112360 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:06.113230 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:06.090003 20568 raft_consensus.cc:1273] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:06.011365 23219 consensus_queue.cc:1035] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20250831 13:59:06.159581 23304 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:06.159801 23184 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:06.161988 20570 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:06.155905 23158 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:06.181456 20568 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1541) took 1012 ms. Trace:
I20250831 13:59:06.201983 20576 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:06.202262 20568 rpcz_store.cc:276] 0831 13:59:05.169116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.089426 (+920310us) service_pool.cc:225] Handling call
0831 13:59:06.089697 (+ 271us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.178810 (+ 89113us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.181381 (+ 2571us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:06.161968 20643 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:06.159996 23204 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.155663 23266 consensus_queue.cc:1046] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20250831 13:59:06.185490 23184 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.215245 20649 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:06.165306 23243 raft_consensus.cc:2802] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250831 13:59:06.163105 23316 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:06.114216 20572 rpcz_store.cc:276] 0831 13:59:04.789555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.991799 (+1202244us) service_pool.cc:225] Handling call
0831 13:59:05.992071 (+ 272us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.113587 (+121516us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.113781 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:06.193761 23216 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20250831 13:59:06.162937 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 343390us
I20250831 13:59:06.200495 23292 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.218863 20570 raft_consensus.cc:1273] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20250831 13:59:06.192978 22987 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:06.198381 23315 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.167156 22819 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:06.232975 20568 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:06.215517 23158 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:06.271485 20651 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:06.212973 23106 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:06.256122 23292 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:06.260079 23159 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.261425 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1579) took 2104 ms. Trace:
I20250831 13:59:06.295907 20573 rpcz_store.cc:276] 0831 13:59:04.156572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.065070 (+908498us) service_pool.cc:225] Handling call
0831 13:59:05.661616 (+596546us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.823899 (+162283us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.261232 (+437333us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b54000b45d0. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:06.261370 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1371099520}
I20250831 13:59:06.297506 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.263892 22987 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:06.266711 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.267563 20466 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:06.269059 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1032) took 1936 ms. Trace:
I20250831 13:59:06.299998 20498 rpcz_store.cc:276] 0831 13:59:04.332757 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.332907 (+ 150us) service_pool.cc:225] Handling call
0831 13:59:06.268888 (+1935981us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400162454. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:06.269005 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2776529792}
I20250831 13:59:06.269852 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 1127) took 1147 ms. Trace:
W20250831 13:59:06.272246 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:06.301151 20650 rpcz_store.cc:276] 0831 13:59:05.122719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.318541 (+195822us) service_pool.cc:225] Handling call
0831 13:59:06.269673 (+951132us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b5400031454. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:06.269799 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1068761472}
I20250831 13:59:06.272138 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.302030 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:06.302568 21208 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:06.302989 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.304054 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.282584 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.287164 23200 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:06.242709 22819 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:06.225803 23243 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:06.295027 20646 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:06.305222 20571 raft_consensus.cc:2391] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:06.313315 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.292541 20575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1539) took 1124 ms. Trace:
I20250831 13:59:06.184940 23313 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.308987 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.302234 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:06.312100 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 44622.9us
I20250831 13:59:06.315104 20616 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.313998 20570 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:06.316869 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.317670 20650 raft_consensus.cc:2391] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:06.318071 20495 raft_consensus.cc:2466] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:06.269570 23277 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.319612 20577 raft_consensus.cc:2391] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:06.322652 20617 leader_election.cc:304] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:06.324811 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.326929 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:06.329715 20616 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.330842 20650 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 8: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:06.331059 20541 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:06.339968 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:06.340407 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.340555 20497 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:59:06.341230 20649 raft_consensus.cc:2391] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:06.331851 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.333503 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:06.342064 20541 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:06.342362 20498 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:06.343230 20541 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:06.345976 20642 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:06.347980 21208 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:06.305644 23204 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.363615 20496 raft_consensus.cc:1238] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 9. Current term is 10. Ops: []
I20250831 13:59:06.366528 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1532) took 1444 ms. Trace:
I20250831 13:59:06.366925 20576 rpcz_store.cc:276] 0831 13:59:04.922320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.025380 (+1103060us) service_pool.cc:225] Handling call
0831 13:59:06.025650 (+ 270us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.202400 (+176750us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.366317 (+163917us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54001fb154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:06.366473 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":405069184}
I20250831 13:59:06.368323 20576 raft_consensus.cc:1273] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:59:06.368432 22371 raft_consensus.cc:3053] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:06.368873 22371 raft_consensus.cc:738] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:06.369750 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.369596 22371 consensus_queue.cc:260] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.370317 20576 raft_consensus.cc:1273] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:59:06.370877 23159 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:06.371443 23159 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:06.371802 23159 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:06.374887 23321 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:06.376760 20541 leader_election.cc:304] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:06.378288 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.379108 20649 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:06.380271 20497 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:59:06.358533 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.359395 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:06.360162 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.315500 23083 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:06.316115 22804 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:06.315974 23246 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:06.316256 23243 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Advancing to term 12
I20250831 13:59:06.370427 20645 raft_consensus.cc:2391] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:06.346621 23224 raft_consensus.cc:2802] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:59:06.357242 20570 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:06.324368 23292 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:06.315850 23200 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.376070 22371 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:06.357815 20571 raft_consensus.cc:1273] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250831 13:59:06.391088 20495 raft_consensus.cc:1273] T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20250831 13:59:06.377352 22987 raft_consensus.cc:2802] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:06.344595 23197 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:06.348821 22999 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:06.348228 23003 raft_consensus.cc:2802] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:06.396593 20617 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:06.451530 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 55030.8us
I20250831 13:59:06.397413 23190 raft_consensus.cc:2802] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:06.399976 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:59:06.402037 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 237792us
I20250831 13:59:06.455412 20649 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:06.387943 20541 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:06.406379 20497 raft_consensus.cc:1273] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:59:06.416842 20498 raft_consensus.cc:2466] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:06.387203 20616 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.385259 22804 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.398917 23266 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:06.455669 20650 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20250831 13:59:06.457579 20498 raft_consensus.cc:1273] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20250831 13:59:06.467850 20498 pending_rounds.cc:85] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 3
I20250831 13:59:06.468089 20498 pending_rounds.cc:107] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20250831 13:59:06.468147 20617 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:06.468396 20498 raft_consensus.cc:2887] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:06.397580 23224 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.383468 23204 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.464826 20649 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
W20250831 13:59:06.465499 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 78405.1us
I20250831 13:59:06.476208 20616 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.398826 23292 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.393122 23246 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.473315 20650 raft_consensus.cc:1238] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250831 13:59:06.391132 22780 consensus_queue.cc:1046] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20250831 13:59:06.418578 23361 consensus_queue.cc:1046] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20250831 13:59:06.467243 23210 raft_consensus.cc:2802] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader election won for term 4
I20250831 13:59:06.459273 22999 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:06.460839 23190 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:06.466799 22804 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:06.400727 23267 consensus_queue.cc:1046] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20250831 13:59:06.467412 23288 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.460614 22987 raft_consensus.cc:695] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [term 9 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:06.466416 23266 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.475023 23224 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:59:06.471185 23197 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 13:59:06.452445 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 51421.6us
I20250831 13:59:06.471709 23003 raft_consensus.cc:695] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:06.478485 23000 raft_consensus.cc:2802] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:06.477723 23083 raft_consensus.cc:2747] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:06.506541 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:06.533568 20498 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:06.484179 23292 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:06.492755 23184 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.505017 22999 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:06.508792 23210 raft_consensus.cc:695] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 4 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:06.469970 23216 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.029s
I20250831 13:59:06.474475 23173 consensus_queue.cc:1035] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
I20250831 13:59:06.488888 23246 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:06.479085 23257 consensus_queue.cc:1046] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20250831 13:59:06.504530 23306 raft_consensus.cc:3053] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:06.508240 23190 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:06.532796 20647 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1451) took 1939 ms. Trace:
W20250831 13:59:06.613297 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 225447us
I20250831 13:59:06.615185 20647 rpcz_store.cc:276] 0831 13:59:04.592987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:04.887123 (+294136us) service_pool.cc:225] Handling call
0831 13:59:04.887397 (+ 274us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:05.770625 (+883228us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.532491 (+761866us) spinlock_profiling.cc:243] Waited 883 ms on lock 0x7b540029f254. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:06.532726 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2029848960}
I20250831 13:59:06.571367 23200 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.510473 23315 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:06.576196 20502 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
W20250831 13:59:06.581311 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:06.598757 23313 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:06.521302 23316 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:59:06.519675 23266 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:06.504340 23321 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20250831 13:59:06.509938 23279 consensus_queue.cc:1046] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.089s
W20250831 13:59:06.626349 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 145769us
W20250831 13:59:06.608891 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 283704us
I20250831 13:59:06.582417 20498 raft_consensus.cc:2466] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:06.599246 23000 raft_consensus.cc:695] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:06.606079 23277 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:06.624235 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 148127us
I20250831 13:59:06.611943 23306 raft_consensus.cc:738] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:06.632411 20616 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:06.633430 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 164280us
I20250831 13:59:06.605646 23372 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20250831 13:59:06.635739 20616 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.638979 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.611929 23184 consensus_queue.cc:1046] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20250831 13:59:06.621268 23316 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.641230 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.639528 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.644881 20569 raft_consensus.cc:2391] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:06.646178 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.645665 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:06.633195 23106 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:06.612169 23374 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20250831 13:59:06.627858 23267 raft_consensus.cc:3053] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:06.653302 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.654551 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.654822 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.653873 20576 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:06.655478 20571 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:06.656044 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.656570 20576 raft_consensus.cc:3058] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:06.651379 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.657974 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.629278 23003 consensus_queue.cc:237] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.640689 23219 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250831 13:59:06.731875 23219 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.732434 23219 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:59:06.670521 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.658957 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.735123 20569 raft_consensus.cc:2391] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:59:06.659564 23368 log.cc:927] Time spent T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20250831 13:59:06.673111 22819 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.660873 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1530) took 1829 ms. Trace:
I20250831 13:59:06.652172 23197 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:06.736442 23197 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:06.655282 23267 raft_consensus.cc:738] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:06.648643 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:06.675734 20647 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:06.737511 22819 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.738250 20495 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:06.738566 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.739315 20647 raft_consensus.cc:2366] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:59:06.740159 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.740666 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.741130 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.666321 23159 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.663105 23125 raft_consensus.cc:3053] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:06.665963 23158 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.743106 20570 raft_consensus.cc:2408] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20250831 13:59:06.744910 23158 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.679229 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.745728 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:06.682356 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:06.683372 23321 raft_consensus.cc:2747] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:59:06.685539 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 49771.4us
I20250831 13:59:06.642973 23210 consensus_queue.cc:237] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.704629 20576 raft_consensus.cc:2466] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:06.737468 21206 raft_consensus.cc:2366] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:59:06.706454 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.760576 20572 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20250831 13:59:06.760991 20572 pending_rounds.cc:85] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 4
I20250831 13:59:06.761258 20572 pending_rounds.cc:107] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20250831 13:59:06.761574 20572 raft_consensus.cc:2887] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:06.709641 23245 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:06.710856 21175 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.764182 20617 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:06.764530 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 53840.6us
I20250831 13:59:06.712143 23000 consensus_queue.cc:237] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:06.718575 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 81245.3us
I20250831 13:59:06.719913 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:06.720470 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.721298 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.766142 20500 raft_consensus.cc:2366] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4.
I20250831 13:59:06.649099 23316 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:06.735023 20568 raft_consensus.cc:2466] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:59:06.659134 23204 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.735998 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.736346 20574 rpcz_store.cc:276] 0831 13:59:04.831452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.006434 (+1174982us) service_pool.cc:225] Handling call
0831 13:59:06.006627 (+ 193us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:06.006810 (+ 183us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:06.006824 (+ 14us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:06.026109 (+ 19285us) log.cc:844] Serialized 26 byte log entry
0831 13:59:06.407922 (+381813us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:06.407989 (+ 67us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.408053 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:06.660670 (+252617us) raft_consensus.cc:1683] finished
0831 13:59:06.660695 (+ 25us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:06.660815 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":319471,"threads_started":1,"wal-append.queue_time_us":431926}
I20250831 13:59:06.770082 20569 raft_consensus.cc:2391] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:06.772292 20501 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:06.743186 20541 leader_election.cc:400] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:59:06.678902 23306 consensus_queue.cc:260] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.779249 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.742224 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.751040 20647 raft_consensus.cc:2391] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:06.752728 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.780771 20495 raft_consensus.cc:2391] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 9: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:06.746513 20570 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:59:06.706176 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.781397 20616 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.763190 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:06.782202 20576 raft_consensus.cc:3058] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:06.783121 20616 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.783911 22901 raft_consensus.cc:2802] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:06.784385 22901 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:06.784783 22901 raft_consensus.cc:3058] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:06.786851 23216 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:06.787405 23216 consensus_queue.cc:1035] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:06.789146 21208 raft_consensus.cc:1238] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:06.763231 20572 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:06.765527 20466 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:06.624773 22987 consensus_queue.cc:237] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.658638 23304 raft_consensus.cc:3053] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:06.736902 23197 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:59:06.771919 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 101907us
I20250831 13:59:06.773144 20541 leader_election.cc:403] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20250831 13:59:06.837360 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 94310us
I20250831 13:59:06.838414 21175 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.758183 23361 consensus_queue.cc:1046] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20250831 13:59:06.794054 23243 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.795106 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.799052 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:06.799417 23378 raft_consensus.cc:2802] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:06.844446 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:06.844632 23378 raft_consensus.cc:695] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:06.845062 20569 raft_consensus.cc:2466] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
W20250831 13:59:06.845994 23360 log.cc:927] Time spent T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.731s user 0.002s sys 0.000s
I20250831 13:59:06.845650 23378 consensus_queue.cc:237] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:06.799697 23382 consensus_queue.cc:1035] T 192560b5dd5040279d722ef31a62c940 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:59:06.846933 23266 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.800817 23292 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.848948 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.849443 20651 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46464 (request call id 102) took 1013 ms. Trace:
I20250831 13:59:06.801177 23387 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:06.850373 23292 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:06.850737 23387 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.802346 20576 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:06.852047 20645 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:06.773600 23313 raft_consensus.cc:2802] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:59:06.820626 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.828562 23321 consensus_queue.cc:1046] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:59:06.743273 23125 raft_consensus.cc:738] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:06.839900 21175 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:06.840143 20543 leader_election.cc:400] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Vote denied by peer ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173) with higher term. Message: Invalid argument: T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
W20250831 13:59:06.884074 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 44246.1us
I20250831 13:59:06.768710 23303 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:06.843191 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.843835 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.833703 23173 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:06.850044 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.819904 20568 raft_consensus.cc:1273] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20250831 13:59:06.849803 20651 rpcz_store.cc:276] 0831 13:59:05.836483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.173009 (+336526us) service_pool.cc:225] Handling call
0831 13:59:06.271312 (+ 98303us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.291421 (+ 20109us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.849272 (+557851us) spinlock_profiling.cc:243] Waited 95 ms on lock 0x7b5400031450. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:06.849392 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":218396416}
I20250831 13:59:06.844110 23243 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.897449 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.898028 20650 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:06.898792 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:06.848896 23266 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:06.835119 23304 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:59:06.842074 22873 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:59:06.905565 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.885082 23306 raft_consensus.cc:3058] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:06.906980 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.797232 22804 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.906111 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.882285 23204 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 44d163094add49049cc749547d891602 (127.18.233.131:45067) with higher term. Message: Invalid argument: T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:59:06.914846 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1653) took 1054 ms. Trace:
I20250831 13:59:06.915243 20499 rpcz_store.cc:276] 0831 13:59:05.860780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.860906 (+ 126us) service_pool.cc:225] Handling call
0831 13:59:05.861241 (+ 335us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:59:05.868996 (+ 7755us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.869026 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:59:05.869569 (+ 543us) log.cc:844] Serialized 52 byte log entry
0831 13:59:05.946958 (+ 77389us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:05.946996 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.947064 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:06.845473 (+898409us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:06.910518 (+ 65045us) raft_consensus.cc:1683] finished
0831 13:59:06.910540 (+ 22us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:06.914782 (+ 4242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7380,"raft.queue_time_us":9717,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":84055808,"thread_start_us":76996,"threads_started":1,"wal-append.queue_time_us":245052}
I20250831 13:59:06.836591 22999 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.923126 23387 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:06.906049 23361 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
W20250831 13:59:06.925309 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:06.853165 23190 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.896551 23273 raft_consensus.cc:2747] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:06.737571 23267 consensus_queue.cc:260] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:06.896965 23173 raft_consensus.cc:491] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:06.897310 23313 raft_consensus.cc:695] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:06.935779 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1135) took 1262 ms. Trace:
W20250831 13:59:06.926649 22371 consensus_meta.cc:287] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.508s user 0.008s sys 0.000s
I20250831 13:59:06.936944 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:06.948568 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 111783us
W20250831 13:59:06.958875 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 173515us
I20250831 13:59:06.952726 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.959205 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1664) took 1084 ms. Trace:
I20250831 13:59:06.960871 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.965772 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.978649 20496 rpcz_store.cc:276] 0831 13:59:05.875091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.980388 (+105297us) service_pool.cc:225] Handling call
0831 13:59:05.980747 (+ 359us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.377651 (+396904us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.958971 (+581320us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54001c3754. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:06.959148 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":880009472}
I20250831 13:59:06.970844 20649 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [term 9 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:06.971604 20646 rpcz_store.cc:276] 0831 13:59:05.673083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.087779 (+414696us) service_pool.cc:225] Handling call
0831 13:59:06.088072 (+ 293us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.313202 (+225130us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.935490 (+622288us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54003a02d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:06.935709 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":475385600}
I20250831 13:59:06.980320 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.957365 20466 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:06.950994 23356 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.806s user 0.002s sys 0.000s
I20250831 13:59:06.949622 22859 raft_consensus.cc:3053] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 10 LEADER]: Stepping down as leader of term 10
I20250831 13:59:06.984112 20577 raft_consensus.cc:1238] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:06.959407 23159 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:07.016850 20646 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20250831 13:59:07.015350 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.986406 20616 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:07.013863 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.971406 23173 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:07.015849 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:07.022111 20645 raft_consensus.cc:1238] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:07.024441 20499 raft_consensus.cc:2391] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 9: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:06.985390 23387 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:07.020108 20616 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:06.982702 23267 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:06.964496 23219 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:06.987135 22859 raft_consensus.cc:738] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:06.932734 23246 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.013836 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.034487 20646 raft_consensus.cc:1273] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20250831 13:59:07.040182 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:59:06.975260 23182 consensus_queue.cc:1162] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ab7ade51fa134a1b814fab748f01d43d" responder_term: 6 status { last_received { term: 5 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250831 13:59:06.975210 22873 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20250831 13:59:06.985633 23003 consensus_queue.cc:1035] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20250831 13:59:07.042438 23243 consensus_queue.cc:1046] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20250831 13:59:07.031065 23387 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:07.033102 23164 consensus_queue.cc:1046] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20250831 13:59:06.984793 23106 consensus_queue.cc:1046] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.051s
I20250831 13:59:07.024902 23264 consensus_queue.cc:1046] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20250831 13:59:07.045683 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:06.932919 23125 consensus_queue.cc:260] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.040907 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.013651 23321 consensus_queue.cc:1035] T bb59414ade0543ef8c7f047104f39576 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20250831 13:59:07.053210 20651 raft_consensus.cc:1238] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:59:07.053911 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1644) took 1530 ms. Trace:
I20250831 13:59:07.072703 20493 rpcz_store.cc:276] 0831 13:59:05.523999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.650460 (+126461us) service_pool.cc:225] Handling call
0831 13:59:05.670254 (+ 19794us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:05.670567 (+ 313us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.670592 (+ 25us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:05.671016 (+ 424us) log.cc:844] Serialized 26 byte log entry
0831 13:59:05.896762 (+225746us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:05.901058 (+ 4296us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.901185 (+ 127us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:06.950312 (+1049127us) spinlock_profiling.cc:243] Waited 6.25 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:07.047965 (+ 97653us) raft_consensus.cc:1683] finished
0831 13:59:07.048012 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:07.053838 (+ 5826us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1345,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":7539,"spinlock_wait_cycles":14377856,"thread_start_us":225400,"threads_started":1,"wal-append.queue_time_us":283974}
I20250831 13:59:06.988055 23224 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.056243 23243 raft_consensus.cc:3053] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:07.057257 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.025394 23000 consensus_queue.cc:1046] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20250831 13:59:07.066565 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.077191 20496 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:07.075446 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.013001 23304 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:07.080016 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.087738 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.087873 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.088799 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.089516 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:07.089987 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 205030us
I20250831 13:59:07.058691 23387 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:07.094830 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.101068 20646 raft_consensus.cc:1238] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:59:07.096101 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.103782 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.047243 23382 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20250831 13:59:07.107883 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.104703 20499 raft_consensus.cc:2466] T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:07.109463 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.079113 23243 raft_consensus.cc:738] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:07.111723 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.112063 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.110644 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:07.164589 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.165654 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:41962 (request call id 12) took 1216 ms. Trace:
I20250831 13:59:07.170799 20644 rpcz_store.cc:276] 0831 13:59:05.942998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.002692 (+ 59694us) service_pool.cc:225] Handling call
0831 13:59:07.120546 (+1117854us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400180c20. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc90ac50 00007fb9bc970612 00007fb9c18d61ba 00007fb9baf701c4 00007fb9baf70152 00007fb9baf70081 00007fb9baf6fffc 00007fb9baf6f2b2 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:07.159552 (+ 39006us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":370652928}
I20250831 13:59:06.322897 20575 rpcz_store.cc:276] 0831 13:59:05.168596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.027727 (+859131us) service_pool.cc:225] Handling call
0831 13:59:06.084446 (+ 56719us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.084669 (+ 223us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 13:59:06.084952 (+ 283us) log.cc:844] Serialized 14 byte log entry
0831 13:59:06.292133 (+207181us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:06.292199 (+ 66us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:06.292267 (+ 68us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.292348 (+ 81us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:06.292474 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83,"prepare.queue_time_us":310,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":11976,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":10821,"spinlock_wait_cycles":31616,"thread_start_us":206830,"threads_started":1,"wal-append.queue_time_us":195501}
I20250831 13:59:07.119257 20493 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:07.104858 23394 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20250831 13:59:07.093622 23316 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:07.183792 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 159613us
I20250831 13:59:07.162892 23381 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:07.120500 22804 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:07.160635 23304 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:07.159152 22999 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:07.080724 23313 consensus_queue.cc:237] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:07.200680 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.205684 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.209262 20644 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:07.175102 23190 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:07.213788 20650 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
W20250831 13:59:07.213922 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 193883us
I20250831 13:59:07.219588 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.204993 23315 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:07.220724 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.221763 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "712d44ed33ec4e37b2bef30d18b3495b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.175720 23392 consensus_queue.cc:1046] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20250831 13:59:07.225019 20500 raft_consensus.cc:2408] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20250831 13:59:07.226053 23224 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:07.227005 21175 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.227346 22804 raft_consensus.cc:3053] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:07.227743 22804 raft_consensus.cc:738] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:07.228554 22804 consensus_queue.cc:260] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.232322 23400 consensus_queue.cc:1035] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:59:07.239496 20493 raft_consensus.cc:2408] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20250831 13:59:07.241524 20616 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:07.202590 23394 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:07.245282 20466 leader_election.cc:304] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:07.243165 23219 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:07.201741 23381 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.272037 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:07.251425 23401 raft_consensus.cc:3053] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Stepping down as leader of term 9
I20250831 13:59:07.253216 22987 raft_consensus.cc:2802] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
W20250831 13:59:07.260162 23391 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20250831 13:59:07.283146 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.300280 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.300500 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.291000 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.291642 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.293761 23125 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:07.277747 23392 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
W20250831 13:59:07.295049 22901 consensus_meta.cc:287] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.510s user 0.005s sys 0.005s
I20250831 13:59:07.296185 23404 consensus_queue.cc:1035] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:59:07.297724 23003 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:07.297981 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.298463 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:07.318989 20576 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:07.302008 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.319931 20575 raft_consensus.cc:3058] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 3 FOLLOWER]: Advancing to term 4
I20250831 13:59:07.256037 23224 consensus_queue.cc:1035] T 223426ea4b53470699bceb4fb06dd992 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20250831 13:59:07.266817 23197 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.303673 23381 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:07.275831 23401 raft_consensus.cc:738] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:07.252643 23378 consensus_queue.cc:1046] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20250831 13:59:07.211719 23243 consensus_queue.cc:260] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.313251 20651 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 9.
I20250831 13:59:07.334662 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:07.307293 23316 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:07.321053 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.322890 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:07.327203 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.330233 20576 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29"
dest_uuid: "212b66976f874aa682cae0fb77928786"
mode: GRACEFUL
new_leader_uuid: "58aded50ab0d4609a053647eb11b1f48"
from {username='slave'} at 127.0.0.1:49676
I20250831 13:59:07.255607 23394 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:59:07.299245 23382 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:59:07.336131 20541 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:07.301453 23313 consensus_queue.cc:1035] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20250831 13:59:07.321585 23404 consensus_queue.cc:1230] T 099d69058d8b46999f57c3142f80acd0 P 44d163094add49049cc749547d891602 [LEADER]: Peer 212b66976f874aa682cae0fb77928786 log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20250831 13:59:07.316207 22987 raft_consensus.cc:695] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:07.345965 20568 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 13:59:07.347239 20645 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:07.347643 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:07.349901 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:07.326859 23392 raft_consensus.cc:695] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:07.366012 20575 raft_consensus.cc:2466] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
I20250831 13:59:07.367218 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.369345 20568 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:07.357210 20645 raft_consensus.cc:1238] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20250831 13:59:07.369904 20651 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:07.318861 22901 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.374104 20651 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:07.319144 23373 consensus_queue.cc:1046] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
W20250831 13:59:07.376355 23359 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 1.232s user 0.007s sys 0.000s
I20250831 13:59:07.377678 22987 consensus_queue.cc:237] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.359526 23381 consensus_queue.cc:1046] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.019s
I20250831 13:59:07.378479 22859 consensus_queue.cc:260] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.379227 23381 consensus_queue.cc:1230] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20250831 13:59:07.349911 23382 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:07.379664 22859 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:59:07.380965 23382 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:07.327387 23003 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.385501 23003 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:07.385926 23401 consensus_queue.cc:260] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.387478 20642 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:07.390821 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:07.393740 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.349720 22999 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:07.395448 20651 raft_consensus.cc:2391] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:07.396193 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:07.398003 21208 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:07.368486 23243 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:07.400684 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1647) took 1835 ms. Trace:
I20250831 13:59:07.402878 20468 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:07.402704 20494 rpcz_store.cc:276] 0831 13:59:05.565182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.691336 (+126154us) service_pool.cc:225] Handling call
0831 13:59:05.691619 (+ 283us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:05.701040 (+ 9421us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:05.701067 (+ 27us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:05.706629 (+ 5562us) log.cc:844] Serialized 26 byte log entry
0831 13:59:05.894606 (+187977us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:05.894648 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:05.894731 (+ 83us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:07.323785 (+1429054us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:07.387303 (+ 63518us) raft_consensus.cc:1683] finished
0831 13:59:07.387342 (+ 39us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:07.400601 (+ 13259us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9083,"raft.queue_time_us":13043,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":4778,"spinlock_wait_cycles":15192448,"thread_start_us":135417,"threads_started":1,"wal-append.queue_time_us":285915}
I20250831 13:59:07.404707 20576 raft_consensus.cc:604] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Received request to transfer leadership to TS 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:07.405054 20576 raft_consensus.cc:612] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:59:07.405121 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.406224 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.407558 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.408740 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.409932 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.411295 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.412158 20494 raft_consensus.cc:2391] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 9: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:07.413146 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.414376 21206 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 13:59:07.415258 23246 leader_election.cc:290] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:07.419695 22901 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:07.420866 20496 raft_consensus.cc:2408] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20250831 13:59:07.424031 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 76491.1us
I20250831 13:59:07.430672 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:07.436044 20396 catalog_manager.cc:5582] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ab7ade51fa134a1b814fab748f01d43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:59:07.436728 22987 consensus_queue.cc:1046] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250831 13:59:07.437189 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 46874.9us
I20250831 13:59:07.425791 20494 raft_consensus.cc:2391] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 10: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:07.384951 23173 raft_consensus.cc:513] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.438710 20568 raft_consensus.cc:2466] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:07.431433 23378 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:07.443288 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:07.443166 23378 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.444608 20616 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:07.445118 23378 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:07.437508 23264 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:07.445804 23315 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:07.445962 23264 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:59:07.446107 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:07.446413 23264 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:07.446848 23399 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:59:07.446988 22901 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:07.447492 22901 raft_consensus.cc:491] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:07.447822 22901 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:07.449496 23173 leader_election.cc:290] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:07.450090 20644 raft_consensus.cc:1238] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:07.450835 20651 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:07.451119 23413 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 13:59:07.423170 23401 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:07.424232 22999 raft_consensus.cc:695] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 9 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:07.440775 23210 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:07.452360 20466 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:07.456215 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.435189 23219 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 13:59:07.457840 20616 leader_election.cc:304] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:07.458429 20494 raft_consensus.cc:2366] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4.
W20250831 13:59:07.442384 23287 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.295s user 0.010s sys 0.032s
I20250831 13:59:07.481524 20568 pending_rounds.cc:85] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 3
I20250831 13:59:07.481943 20568 pending_rounds.cc:107] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20250831 13:59:07.482669 20568 raft_consensus.cc:2887] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:07.496318 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:07.497359 22371 raft_consensus.cc:2802] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader election won for term 10
I20250831 13:59:07.498152 22371 raft_consensus.cc:695] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 10 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:07.487004 22987 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.012s
I20250831 13:59:07.499152 22371 consensus_queue.cc:237] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.497383 23224 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:07.500721 23224 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.502732 23224 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:07.505788 23411 log.cc:927] Time spent T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20250831 13:59:07.474100 23373 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:59:07.506577 23373 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:07.506945 23373 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:07.418509 23392 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:07.508416 23197 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:07.618526 23381 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:59:07.654443 23418 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20250831 13:59:07.870303 23381 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:07.871522 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:07.872206 20570 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:07.873127 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:07.813380 20501 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
W20250831 13:59:07.623039 23419 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20250831 13:59:07.873703 20570 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
W20250831 13:59:07.754838 23422 log.cc:927] Time spent T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
W20250831 13:59:07.651643 23414 log.cc:927] Time spent T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
I20250831 13:59:07.874994 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.584224 20568 pending_rounds.cc:107] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20250831 13:59:07.875927 20570 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:07.876109 20568 raft_consensus.cc:2887] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:07.875847 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.584867 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:59:07.711412 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.805466 23264 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:07.654203 23106 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:07.877058 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:07.575850 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.908699 20575 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:07.910811 20651 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20250831 13:59:07.869136 23366 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 1.588s user 0.002s sys 0.011s
I20250831 13:59:07.912966 20651 raft_consensus.cc:1238] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
W20250831 13:59:07.477258 23287 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.330s user 0.010s sys 0.032s
W20250831 13:59:07.787633 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:07.782158 21175 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:07.663704 23267 consensus_meta.cc:287] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.624s user 0.003s sys 0.006s
I20250831 13:59:07.921056 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:07.925673 20570 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:59:07.660863 23387 consensus_meta.cc:287] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.554s user 0.008s sys 0.000s
I20250831 13:59:07.480510 23313 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:07.923820 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:07.619707 23412 log.cc:927] Time spent T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
W20250831 13:59:07.929260 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:07.929776 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.931387 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:07.584494 22987 raft_consensus.cc:491] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:07.944047 20651 raft_consensus.cc:1215] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 LEADER]: Deduplicated request from leader. Original: 3.2->[5.3-5.3] Dedup: 5.3->[]
I20250831 13:59:07.944209 20645 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:07.944591 20651 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20250831 13:59:07.945406 20541 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:07.945973 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:07.464148 23210 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.510325 23394 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
I20250831 13:59:07.949285 20498 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:07.924800 23304 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:07.767953 23373 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:07.952329 20617 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:07.972970 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:07.979948 20575 raft_consensus.cc:2466] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:08.000948 23427 raft_consensus.cc:491] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:07.979244 23381 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:07.918498 23106 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.868263 22901 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.659984 22999 consensus_queue.cc:237] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:07.949311 22859 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:59:08.005915 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 60582.2us
I20250831 13:59:08.010530 20570 raft_consensus.cc:2466] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:07.944537 23313 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.011799 20543 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:08.001861 23381 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:08.011685 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:08.013783 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:07.979568 23267 consensus_queue.cc:1046] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20250831 13:59:07.952711 23374 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.057s
I20250831 13:59:08.015756 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:07.951632 22987 raft_consensus.cc:513] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.001485 23427 raft_consensus.cc:513] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.018617 20571 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:59:08.021945 23433 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 13:59:08.021950 23264 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:07.998754 23304 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:07.945655 23387 raft_consensus.cc:513] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.013743 22859 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.020602 23381 raft_consensus.cc:695] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [term 7 LEADER]: Becoming Leader. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:08.030546 22987 leader_election.cc:290] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:08.032727 23417 log.cc:927] Time spent T 7562a669789845418660f6f1c3109b1e P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.368s user 0.002s sys 0.011s
I20250831 13:59:08.034806 23432 consensus_queue.cc:1046] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20250831 13:59:08.037864 23432 consensus_queue.cc:1230] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20250831 13:59:08.037860 23371 log.cc:927] Time spent T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602: Append to log took a long time: real 1.752s user 0.006s sys 0.016s
I20250831 13:59:08.038420 23106 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.040791 23373 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.040799 22859 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:08.041316 23396 log.cc:927] Time spent T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.834s user 0.006s sys 0.016s
I20250831 13:59:08.041720 23313 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:08.042367 23398 log.cc:927] Time spent T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.786s user 0.008s sys 0.013s
I20250831 13:59:08.042582 23427 leader_election.cc:290] T f29595dd2c044618bd6cb4517fbc5c7b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.042646 23210 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:08.043249 22901 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:08.046737 23387 leader_election.cc:290] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.046788 23303 raft_consensus.cc:3053] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:08.047402 23303 raft_consensus.cc:738] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
W20250831 13:59:08.047528 23420 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.383s user 0.000s sys 0.011s
W20250831 13:59:08.048242 23367 log.cc:927] Time spent T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602: Append to log took a long time: real 1.769s user 0.009s sys 0.008s
I20250831 13:59:08.048302 23303 consensus_queue.cc:260] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:08.049450 23303 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:59:08.053794 23369 log.cc:927] Time spent T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 1.765s user 0.003s sys 0.011s
I20250831 13:59:08.055526 23304 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:08.055908 23381 consensus_queue.cc:237] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.011592 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:08.093391 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 143162us
I20250831 13:59:08.094027 20494 raft_consensus.cc:2391] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 6: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:08.095566 20499 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 9: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:08.098239 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.099767 20494 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:08.101231 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.102276 20494 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:08.103394 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:08.105993 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.111649 23259 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:08.112597 23259 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:59:08.113153 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:08.113780 20643 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1504) took 2249 ms. Trace:
I20250831 13:59:08.109172 20572 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20250831 13:59:08.114308 20643 rpcz_store.cc:276] 0831 13:59:05.864686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.206639 (+341953us) service_pool.cc:225] Handling call
0831 13:59:06.208143 (+ 1504us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:06.208757 (+ 614us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:06.208799 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:06.209330 (+ 531us) log.cc:844] Serialized 26 byte log entry
0831 13:59:06.285422 (+ 76092us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:06.285460 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.285524 (+ 64us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:08.037309 (+1751785us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:08.113165 (+ 75856us) raft_consensus.cc:1683] finished
0831 13:59:08.113216 (+ 51us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.113557 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":641,"raft.queue_time_us":88,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":12589,"spinlock_wait_cycles":41540864,"thread_start_us":75757,"threads_started":1,"wal-append.queue_time_us":76039}
I20250831 13:59:08.116299 23259 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.106874 20494 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:08.116921 20572 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 9. Current term is 10. Ops: [7.4-9.5]
I20250831 13:59:08.119282 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1689) took 1351 ms. Trace:
I20250831 13:59:08.120550 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1056) took 1201 ms. Trace:
I20250831 13:59:08.119737 20493 rpcz_store.cc:276] 0831 13:59:06.767618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:07.240257 (+472639us) service_pool.cc:225] Handling call
0831 13:59:07.240650 (+ 393us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:59:07.248487 (+ 7837us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:07.248515 (+ 28us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:59:07.249087 (+ 572us) log.cc:844] Serialized 52 byte log entry
0831 13:59:07.337859 (+ 88772us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:07.337896 (+ 37us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:07.337964 (+ 68us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:08.099946 (+761982us) raft_consensus.cc:1683] finished
0831 13:59:08.100458 (+ 512us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.119016 (+ 18558us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11555,"raft.queue_time_us":7635,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":5075,"thread_start_us":76004,"threads_started":1,"wal-append.queue_time_us":163875}
I20250831 13:59:08.121310 23197 raft_consensus.cc:491] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:08.122710 20645 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:08.122853 20648 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1499) took 2451 ms. Trace:
I20250831 13:59:08.120985 20498 rpcz_store.cc:276] 0831 13:59:06.918626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.979296 (+ 60670us) service_pool.cc:225] Handling call
0831 13:59:08.120128 (+1140832us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54000127d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:08.120347 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2139855232}
I20250831 13:59:08.124130 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:08.124881 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.125265 20645 raft_consensus.cc:2391] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 7: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:08.126473 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.123276 20648 rpcz_store.cc:276] 0831 13:59:05.646784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.005022 (+358238us) service_pool.cc:225] Handling call
0831 13:59:06.005318 (+ 296us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:06.023362 (+ 18044us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:06.023396 (+ 34us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:06.023821 (+ 425us) log.cc:844] Serialized 26 byte log entry
0831 13:59:06.280987 (+257166us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:06.281050 (+ 63us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.281127 (+ 77us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:08.045457 (+1764330us) spinlock_profiling.cc:243] Waited 26.9 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:08.097835 (+ 52378us) raft_consensus.cc:1683] finished
0831 13:59:08.097881 (+ 46us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.098335 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17663,"raft.queue_time_us":27531,"raft.run_cpu_time_us":1057,"raft.run_wall_time_us":4643,"spinlock_wait_cycles":61809408,"thread_start_us":168928,"threads_started":1,"wal-append.queue_time_us":255440}
I20250831 13:59:08.129539 23259 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:08.129657 20502 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1662) took 2255 ms (client timeout 2122 ms). Trace:
I20250831 13:59:08.128110 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:08.130196 20502 rpcz_store.cc:269] 0831 13:59:05.874567 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:05.922742 (+ 48175us) service_pool.cc:225] Handling call
0831 13:59:08.129275 (+2206533us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54000122d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:08.129454 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":849852288}
I20250831 13:59:08.122337 23197 raft_consensus.cc:513] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.131661 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.131831 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.131951 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1627) took 2234 ms. Trace:
I20250831 13:59:08.132424 20573 rpcz_store.cc:276] 0831 13:59:05.897821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.327249 (+429428us) service_pool.cc:225] Handling call
0831 13:59:06.327669 (+ 420us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:06.454766 (+127097us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:06.454816 (+ 50us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:06.455344 (+ 528us) log.cc:844] Serialized 26 byte log entry
0831 13:59:06.629390 (+174046us) raft_consensus.cc:1643] Marking committed up to 6
0831 13:59:06.629445 (+ 55us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.131286 (+1501841us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54002a8fd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:08.131381 (+ 95us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:08.131452 (+ 71us) raft_consensus.cc:1683] finished
0831 13:59:08.131467 (+ 15us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.131777 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":209,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":10472,"spinlock_wait_cycles":291129984,"thread_start_us":148759,"threads_started":1,"wal-append.queue_time_us":236586,"wal-append.run_cpu_time_us":1561,"wal-append.run_wall_time_us":1007725}
I20250831 13:59:08.163213 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.135166 23259 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.138692 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:08.139072 20570 raft_consensus.cc:1238] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
W20250831 13:59:08.140340 20647 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 1163) took 1431 ms (client timeout 1881 ms). Trace:
I20250831 13:59:08.144577 20643 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:08.149953 23436 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:08.151885 22819 raft_consensus.cc:3053] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:08.156669 23279 raft_consensus.cc:3053] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:08.158963 20494 raft_consensus.cc:2408] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 13:59:08.158876 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:08.160029 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:08.134191 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.174391 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.174511 20645 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:08.175972 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.016531 20543 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:08.176923 20494 raft_consensus.cc:2366] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 9. Current term is 10.
I20250831 13:59:08.178717 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:08.179855 20651 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:59:08.166356 20647 rpcz_store.cc:269] 0831 13:59:06.708557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.740925 (+ 32368us) service_pool.cc:225] Handling call
0831 13:59:08.139826 (+1398901us) spinlock_profiling.cc:243] Waited 9.03 ms on lock 0x7b5400372154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:08.140109 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20766592}
I20250831 13:59:08.181020 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.184597 22804 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:08.189966 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:08.191453 23266 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:59:08.191915 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 53571.5us
I20250831 13:59:08.193157 20617 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:08.194434 23421 log.cc:927] Time spent T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.531s user 0.007s sys 0.000s
I20250831 13:59:08.172976 22819 raft_consensus.cc:738] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:08.173151 23279 raft_consensus.cc:738] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:08.214982 21208 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:08.218559 23245 raft_consensus.cc:3053] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:59:08.218994 23245 raft_consensus.cc:738] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:08.220201 23245 consensus_queue.cc:260] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.204401 23266 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20250831 13:59:08.224267 21176 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.218580 23279 consensus_queue.cc:260] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.226125 20642 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57702 (request call id 1164) took 1269 ms. Trace:
I20250831 13:59:08.226720 23259 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.227285 20502 raft_consensus.cc:2466] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 8.
I20250831 13:59:08.230554 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.231493 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.235052 23381 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250831 13:59:08.236358 23387 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:08.237250 23387 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.239852 22819 consensus_queue.cc:260] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.240965 23386 raft_consensus.cc:3053] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:08.241268 22819 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:08.241606 20494 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:08.241451 23386 raft_consensus.cc:738] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:08.242647 23000 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:08.242892 23386 consensus_queue.cc:260] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.243412 23000 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:08.244341 23386 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:08.246006 23000 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:08.184648 23436 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.247218 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c719f2a4c3ea4418937337e8e3dd2164" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.247506 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1686) took 1480 ms. Trace:
I20250831 13:59:08.247970 20502 raft_consensus.cc:2466] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 4.
I20250831 13:59:08.247874 20499 rpcz_store.cc:276] 0831 13:59:06.767045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:07.110405 (+343360us) service_pool.cc:225] Handling call
0831 13:59:08.247241 (+1136836us) spinlock_profiling.cc:243] Waited 922 ms on lock 0x7b5400162454. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:08.247441 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2120955136}
I20250831 13:59:08.249352 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.249262 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.251396 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:08.234300 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 217687us
I20250831 13:59:08.252642 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.253379 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.253669 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1043) took 1531 ms. Trace:
I20250831 13:59:08.257772 23190 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:08.263188 20396 catalog_manager.cc:5582] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ab7ade51fa134a1b814fab748f01d43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250831 13:59:08.226498 20642 rpcz_store.cc:276] 0831 13:59:06.956309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.975937 (+ 19628us) service_pool.cc:225] Handling call
0831 13:59:08.225731 (+1249794us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b5400283054. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:08.225970 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":840031488}
I20250831 13:59:08.287772 23436 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:08.288586 23388 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 1.269s user 0.004s sys 0.016s
I20250831 13:59:08.283123 20496 rpcz_store.cc:276] 0831 13:59:06.721642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:07.094583 (+372941us) service_pool.cc:225] Handling call
0831 13:59:08.253327 (+1158744us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54001c2854. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:08.253500 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":744727936}
I20250831 13:59:08.290787 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1681) took 1996 ms. Trace:
I20250831 13:59:08.290963 20493 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:08.291347 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57702 (request call id 1159) took 1964 ms. Trace:
I20250831 13:59:08.291764 20649 rpcz_store.cc:276] 0831 13:59:06.326936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.475300 (+148364us) service_pool.cc:225] Handling call
0831 13:59:06.475603 (+ 303us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.980160 (+504557us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.290862 (+1310702us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54002eb154. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:08.291153 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1138340096}
I20250831 13:59:08.292476 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.291242 20497 rpcz_store.cc:276] 0831 13:59:06.294229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.456575 (+162346us) service_pool.cc:225] Handling call
0831 13:59:06.483263 (+ 26688us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:06.483991 (+ 728us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:06.484034 (+ 43us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:06.484536 (+ 502us) log.cc:844] Serialized 26 byte log entry
0831 13:59:06.883334 (+398798us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:06.883372 (+ 38us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:06.883433 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:08.281466 (+1398033us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b440002304c. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99262b5 00007fb9b9926cd2 00007fb9bc5c6231 00007fb9bc5c1b5a 00007fb9bc5b591e 00007fb9bc5b4c66 00007fb9bc5b4888 00007fb9bc5be2c1 00007fb9bc5be279 00007fb9bc5be209
0831 13:59:08.290273 (+ 8807us) raft_consensus.cc:1683] finished
0831 13:59:08.290293 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.290586 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2652,"raft.queue_time_us":130034,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":7183,"spinlock_wait_cycles":6482304,"thread_start_us":398452,"threads_started":1,"wal-append.queue_time_us":488225}
I20250831 13:59:08.293352 20496 raft_consensus.cc:2391] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 8: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:08.293593 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.294261 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.268313 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.268518 20644 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1528) took 1175 ms. Trace:
I20250831 13:59:08.268806 23245 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
W20250831 13:59:08.252702 20501 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1047) took 1529 ms (client timeout 1999 ms). Trace:
W20250831 13:59:08.251580 20500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1040) took 1530 ms (client timeout 1780 ms). Trace:
I20250831 13:59:08.272740 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:08.273990 20651 raft_consensus.cc:1238] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:08.274977 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:08.275827 20617 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:08.278882 23159 raft_consensus.cc:3053] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:08.250653 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.294792 23370 raft_consensus.cc:3053] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:08.295006 23190 raft_consensus.cc:513] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } attrs { replace: true } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:08.294301 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1553) took 1647 ms. Trace:
I20250831 13:59:08.296187 23197 leader_election.cc:290] T 099d69058d8b46999f57c3142f80acd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:08.298455 23363 log.cc:927] Time spent T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 2.018s user 0.016s sys 0.003s
I20250831 13:59:08.300410 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:08.301192 20650 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1529) took 1057 ms. Trace:
I20250831 13:59:08.292091 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e93846478efb47a1b008fc73d5d25914" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:59:08.302707 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.289732 23279 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:08.267189 20646 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1525) took 1413 ms. Trace:
I20250831 13:59:08.305222 20498 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:08.309157 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1080) took 1006 ms. Trace:
I20250831 13:59:08.341742 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.305711 20644 rpcz_store.cc:276] 0831 13:59:07.093031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:07.210261 (+117230us) service_pool.cc:225] Handling call
0831 13:59:07.210584 (+ 323us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:07.450453 (+239869us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.267953 (+817500us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54000307d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:08.268332 (+ 379us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":550363520}
I20250831 13:59:08.363329 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.319381 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.317338 23000 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:08.363538 20573 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 8.
I20250831 13:59:08.364753 20496 raft_consensus.cc:2366] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9.
W20250831 13:59:08.343822 21175 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.309633 20647 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:08.366456 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.367449 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1094) took 1032 ms. Trace:
W20250831 13:59:08.306252 20501 rpcz_store.cc:269] 0831 13:59:06.722638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.888554 (+165916us) service_pool.cc:225] Handling call
0831 13:59:08.252337 (+1363783us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b540021cad4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:08.252502 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2070668672}
I20250831 13:59:08.365737 23370 raft_consensus.cc:738] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:08.369645 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.370467 20501 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:08.363467 23378 consensus_queue.cc:1046] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20250831 13:59:08.371012 20651 raft_consensus.cc:1215] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 LEADER]: Deduplicated request from leader. Original: 3.2->[5.3-5.3] Dedup: 5.3->[]
I20250831 13:59:08.371637 20651 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [term 6 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20250831 13:59:08.367851 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 92277.5us
I20250831 13:59:08.369995 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:08.372076 20642 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d"
dest_uuid: "44d163094add49049cc749547d891602"
mode: GRACEFUL
new_leader_uuid: "58aded50ab0d4609a053647eb11b1f48"
from {username='slave'} at 127.0.0.1:41962
I20250831 13:59:08.371640 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:08.372830 20642 raft_consensus.cc:604] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 4 LEADER]: Received request to transfer leadership to TS 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:08.373262 20501 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 9: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:08.374109 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.323237 20499 raft_consensus.cc:2366] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 9. Current term is 10.
I20250831 13:59:08.368026 23315 raft_consensus.cc:2747] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250831 13:59:08.313486 20648 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:08.367070 20577 rpcz_store.cc:276] 0831 13:59:06.646461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.659708 (+ 13247us) service_pool.cc:225] Handling call
0831 13:59:06.660015 (+ 307us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:06.985982 (+325967us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.293841 (+1307859us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54003d40d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:08.294140 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":744961920}
I20250831 13:59:08.372104 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:08.364735 20646 rpcz_store.cc:276] 0831 13:59:06.854042 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:07.040710 (+186668us) service_pool.cc:225] Handling call
0831 13:59:07.052262 (+ 11552us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:07.101560 (+ 49298us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.266616 (+1165056us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b540028c654. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:08.266986 (+ 370us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111632256}
I20250831 13:59:08.375996 20617 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:08.363538 20497 raft_consensus.cc:2374] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 44d163094add49049cc749547d891602 in term 6. Re-sending same reply.
I20250831 13:59:08.377312 20493 rpcz_store.cc:276] 0831 13:59:07.335304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.366190 (+1030886us) service_pool.cc:225] Handling call
0831 13:59:08.367377 (+ 1187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.364486 20498 raft_consensus.cc:2391] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:08.362521 20494 rpcz_store.cc:276] 0831 13:59:07.303095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.200658 (+897563us) service_pool.cc:225] Handling call
0831 13:59:08.309105 (+108447us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.308372 23159 raft_consensus.cc:738] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:08.319957 23377 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20250831 13:59:08.387890 20502 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
W20250831 13:59:08.306615 20500 rpcz_store.cc:269] 0831 13:59:06.720897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:06.769687 (+ 48790us) service_pool.cc:225] Handling call
0831 13:59:08.251257 (+1481570us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54002df854. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:08.251443 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":881782528}
I20250831 13:59:08.365329 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52980 (request call id 1079) took 1062 ms. Trace:
I20250831 13:59:08.390466 20496 rpcz_store.cc:276] 0831 13:59:07.302790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.319065 (+1016275us) service_pool.cc:225] Handling call
0831 13:59:08.365224 (+ 46159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.392987 20645 raft_consensus.cc:2466] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 10.
I20250831 13:59:08.312590 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:08.310837 23387 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.399030 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:08.399003 20468 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:08.378088 20650 rpcz_store.cc:276] 0831 13:59:07.243884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:07.246606 (+ 2722us) service_pool.cc:225] Handling call
0831 13:59:07.246996 (+ 390us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:07.270534 (+ 23538us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:07.270579 (+ 45us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:08.142571 (+871992us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b2c0003ebe8. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bb3199a1 00007fb9bb328fd2 00007fb9bc9b2021 00007fb9bc9763fa 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32
0831 13:59:08.143364 (+ 793us) log.cc:844] Serialized 26 byte log entry
0831 13:59:08.203168 (+ 59804us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:08.205496 (+ 2328us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.222163 (+ 16667us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:08.282511 (+ 60348us) raft_consensus.cc:1683] finished
0831 13:59:08.282529 (+ 18us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.300938 (+ 18409us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23089,"raft.queue_time_us":23644,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"spinlock_wait_cycles":7580032,"thread_start_us":59034,"threads_started":1,"wal-append.queue_time_us":103328}
I20250831 13:59:08.402391 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:08.403020 21208 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:08.400388 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:08.404848 21175 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:08.365363 23000 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:08.401688 23315 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:08.411129 19366 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20250831 13:59:08.416803 20571 raft_consensus.cc:3058] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:59:08.418597 20499 raft_consensus.cc:1238] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20250831 13:59:08.419525 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 1068) took 1497 ms. Trace:
I20250831 13:59:08.419852 20499 rpcz_store.cc:276] 0831 13:59:06.922413 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.398765 (+1476352us) service_pool.cc:225] Handling call
0831 13:59:08.418205 (+ 19440us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:59:08.419149 (+ 944us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.419438 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:59:08.420579 20397 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59926 (request call id 134) took 18447 ms (client timeout 15000 ms). Trace:
I20250831 13:59:08.421459 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1696) took 1396 ms. Trace:
W20250831 13:59:08.489643 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.432497 20502 raft_consensus.cc:1238] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250831 13:59:08.434141 20493 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:08.479912 20496 raft_consensus.cc:1238] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:08.428777 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:08.427243 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.428900 20568 raft_consensus.cc:1273] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:08.433982 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1691) took 1666 ms. Trace:
I20250831 13:59:08.421996 20499 rpcz_store.cc:276] 0831 13:59:07.025038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.420654 (+1395616us) service_pool.cc:225] Handling call
0831 13:59:08.420949 (+ 295us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.421134 (+ 185us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:08.421156 (+ 22us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:08.421225 (+ 69us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:08.421254 (+ 29us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.421301 (+ 47us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.421396 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.470717 20647 raft_consensus.cc:2391] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 5: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:08.510847 20497 rpcz_store.cc:276] 0831 13:59:06.767912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.394929 (+1627017us) service_pool.cc:225] Handling call
0831 13:59:08.432964 (+ 38035us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.433355 (+ 391us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:08.433414 (+ 59us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:08.433503 (+ 89us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:08.433569 (+ 66us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.433679 (+ 110us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.433853 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250831 13:59:08.478648 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 62483.8us
W20250831 13:59:08.422812 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.426008 20466 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:08.428256 20575 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 13:59:08.423614 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1692) took 1401 ms. Trace:
I20250831 13:59:08.433593 19366 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250831 13:59:08.524246 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.519518 20500 rpcz_store.cc:276] 0831 13:59:07.022568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.400994 (+1378426us) service_pool.cc:225] Handling call
0831 13:59:08.415485 (+ 14491us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.423238 (+ 7753us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:08.423284 (+ 46us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:08.423348 (+ 64us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:08.423388 (+ 40us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.423442 (+ 54us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.423561 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1003264}
I20250831 13:59:08.412140 23315 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:08.525167 20493 raft_consensus.cc:1238] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
W20250831 13:59:08.531162 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.433946 20616 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:08.521409 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 1069) took 1595 ms. Trace:
I20250831 13:59:08.535398 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1698) took 1340 ms. Trace:
I20250831 13:59:08.522248 20496 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52980 (request call id 1062) took 1589 ms. Trace:
I20250831 13:59:08.535910 20502 rpcz_store.cc:276] 0831 13:59:06.966248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.397113 (+1430865us) service_pool.cc:225] Handling call
0831 13:59:08.432217 (+ 35104us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:08.517770 (+ 85553us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.518024 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.536337 20496 rpcz_store.cc:276] 0831 13:59:06.920563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.396304 (+1475741us) service_pool.cc:225] Handling call
0831 13:59:08.479647 (+ 83343us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.509656 (+ 30009us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.509909 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2432}
I20250831 13:59:08.537320 20493 rpcz_store.cc:276] 0831 13:59:07.195146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.520574 (+1325428us) service_pool.cc:225] Handling call
0831 13:59:08.524993 (+ 4419us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.535168 (+ 10175us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.535348 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.534157 22819 raft_consensus.cc:491] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:08.515077 23387 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
W20250831 13:59:08.538479 20543 leader_election.cc:336] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.092s (SENT)
I20250831 13:59:08.539301 20571 raft_consensus.cc:2466] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:08.540753 22819 raft_consensus.cc:513] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:08.542918 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 116996us
I20250831 13:59:08.561303 20466 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:08.543676 23264 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:08.561990 23264 raft_consensus.cc:491] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:59:08.562657 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 129055us
I20250831 13:59:08.562381 23264 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:59:08.555356 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 137599us
I20250831 13:59:08.425671 23435 consensus_queue.cc:1046] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.133s
I20250831 13:59:08.539196 23315 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:59:08.558606 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.434157 23386 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:08.421000 20397 rpcz_store.cc:269] 0831 13:58:49.973530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:49.988405 (+ 14875us) service_pool.cc:225] Handling call
0831 13:59:08.420346 (+18431941us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:59:00.040664 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:59:00.091264 (+ 50600us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:59:00.106052 (+ 14788us) write_op.cc:435] Acquired schema lock
0831 13:59:00.115182 (+ 9130us) tablet.cc:660] Decoding operations
0831 13:59:00.552966 (+437784us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:59:00.563126 (+ 10160us) write_op.cc:641] Partition lock acquired for write op
0831 13:59:00.585554 (+ 22428us) tablet.cc:683] Acquiring locks for 30 operations
0831 13:59:00.631245 (+ 45691us) tablet.cc:699] Row locks acquired
0831 13:59:00.631287 (+ 42us) write_op.cc:260] PREPARE: finished
0831 13:59:00.631506 (+ 219us) write_op.cc:270] Start()
0831 13:59:00.631702 (+ 196us) write_op.cc:276] Timestamp: P: 1756648740631425 usec, L: 0
0831 13:59:00.631725 (+ 23us) op_driver.cc:348] REPLICATION: starting
0831 13:59:00.669236 (+ 37511us) log.cc:844] Serialized 11928 byte log entry
0831 13:59:01.719074 (+1049838us) op_driver.cc:464] REPLICATION: finished
0831 13:59:02.049535 (+330461us) write_op.cc:301] APPLY: starting
0831 13:59:02.049635 (+ 100us) tablet.cc:1365] starting BulkCheckPresence
0831 13:59:02.050173 (+ 538us) tablet.cc:1368] finished BulkCheckPresence
0831 13:59:02.050307 (+ 134us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:59:02.170479 (+120172us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:59:02.181499 (+ 11020us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=30
0831 13:59:02.181581 (+ 82us) write_op.cc:312] APPLY: finished
0831 13:59:02.401380 (+219799us) log.cc:844] Serialized 199 byte log entry
0831 13:59:02.507290 (+105910us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:59:02.707575 (+200285us) write_op.cc:454] Released schema lock
0831 13:59:02.823677 (+116102us) write_op.cc:341] FINISH: Updating metrics
0831 13:59:02.987057 (+163380us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:59:03.007086 (+ 20029us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1082020,"reported_tablets":54,"child_traces":[["op",{"apply.queue_time_us":321100,"apply.run_cpu_time_us":8542,"apply.run_wall_time_us":857181,"num_ops":30,"prepare.queue_time_us":224696,"prepare.run_cpu_time_us":54486,"prepare.run_wall_time_us":885104,"raft.queue_time_us":593846,"raft.run_cpu_time_us":44606,"raft.run_wall_time_us":302671,"replication_time_us":1061559,"thread_start_us":1011664,"threads_started":4,"wal-append.queue_time_us":363620,"wal-append.run_cpu_time_us":25781,"wal-append.run_wall_time_us":3481979}]]}
I20250831 13:59:08.563902 20616 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:08.440155 22873 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:08.477001 23190 leader_election.cc:290] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:08.444166 23000 leader_election.cc:290] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:08.569556 20501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1695) took 1543 ms. Trace:
W20250831 13:59:08.571468 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.701061 23386 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.562388 22819 leader_election.cc:290] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.727931 20493 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:08.693863 20496 raft_consensus.cc:1273] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:59:08.700328 23392 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:08.683549 20499 raft_consensus.cc:1273] T f29595dd2c044618bd6cb4517fbc5c7b P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20250831 13:59:08.736446 20617 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:08.736852 20497 raft_consensus.cc:1238] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 11. Current term is 12. Ops: []
I20250831 13:59:08.737263 20499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1701) took 1462 ms. Trace:
I20250831 13:59:08.737748 20499 rpcz_store.cc:276] 0831 13:59:07.274798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.537202 (+1262404us) service_pool.cc:225] Handling call
0831 13:59:08.570292 (+ 33090us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.732347 (+162055us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.737199 (+ 4852us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.740275 20497 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1703) took 1384 ms. Trace:
I20250831 13:59:08.740669 20497 rpcz_store.cc:276] 0831 13:59:07.355811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.533854 (+1178043us) service_pool.cc:225] Handling call
0831 13:59:08.726907 (+193053us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.740022 (+ 13115us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.740221 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.700373 20501 rpcz_store.cc:276] 0831 13:59:07.023795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.403847 (+1380052us) service_pool.cc:225] Handling call
0831 13:59:08.411188 (+ 7341us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.411575 (+ 387us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:59:08.418089 (+ 6514us) log.cc:844] Serialized 14 byte log entry
0831 13:59:08.566979 (+148890us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:08.567059 (+ 80us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:08.567119 (+ 60us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.567202 (+ 83us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.567357 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40767,"prepare.queue_time_us":213,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":54,"raft.queue_time_us":45966,"thread_start_us":105099,"threads_started":1}
I20250831 13:59:08.427313 23159 consensus_queue.cc:260] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.577163 22873 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:08.795209 22873 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:08.737322 23392 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:08.799005 23392 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:59:08.802772 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 75342.2us
I20250831 13:59:08.803562 20466 leader_election.cc:304] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:08.807264 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:08.808320 20573 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:08.808405 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:08.809163 20571 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:08.781524 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250831 13:59:08.810078 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:08.810714 20568 raft_consensus.cc:3058] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:08.814251 20576 raft_consensus.cc:3053] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:08.814589 20576 raft_consensus.cc:738] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:08.815322 20500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1707) took 1345 ms. Trace:
I20250831 13:59:08.815464 20576 consensus_queue.cc:260] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.815881 20500 rpcz_store.cc:276] 0831 13:59:07.469497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.534032 (+1064535us) service_pool.cc:225] Handling call
0831 13:59:08.567448 (+ 33416us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.814799 (+247351us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:08.814847 (+ 48us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:08.814912 (+ 65us) raft_consensus.cc:1643] Marking committed up to 7
0831 13:59:08.814973 (+ 61us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.815058 (+ 85us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.815219 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.817693 20570 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250831 13:59:08.782966 20467 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:08.783555 20497 raft_consensus.cc:1273] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250831 13:59:08.820492 23386 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:08.571044 23308 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:08.821497 23308 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.824134 23308 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:08.824728 23308 raft_consensus.cc:2802] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election won for term 10
I20250831 13:59:08.825275 23308 raft_consensus.cc:695] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 10 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:08.826352 20568 raft_consensus.cc:2466] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
W20250831 13:59:08.787418 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 254075us
I20250831 13:59:08.789126 20493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:52966 (request call id 511) took 2020 ms. Trace:
I20250831 13:59:08.568127 23387 raft_consensus.cc:695] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:08.781380 20496 raft_consensus.cc:1238] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:08.786088 22819 consensus_queue.cc:1046] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20250831 13:59:08.739022 22804 consensus_meta.cc:287] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.554s user 0.009s sys 0.005s
I20250831 13:59:08.816793 20576 raft_consensus.cc:3058] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:08.845801 20493 rpcz_store.cc:276] 0831 13:59:06.768473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.787978 (+2019505us) service_pool.cc:225] Handling call
0831 13:59:08.789062 (+ 1084us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250831 13:59:08.735680 23378 raft_consensus.cc:491] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:08.827179 20572 raft_consensus.cc:1273] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250831 13:59:08.741884 23377 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20250831 13:59:08.853196 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 285388us
I20250831 13:59:08.855934 20643 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:08.853286 23377 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:08.881881 23402 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:08.848430 23381 consensus_queue.cc:1046] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20250831 13:59:08.826460 23308 consensus_queue.cc:237] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.887537 23000 raft_consensus.cc:2802] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
W20250831 13:59:08.907846 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 344038us
I20250831 13:59:08.910833 23083 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:59:08.911315 23083 raft_consensus.cc:491] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:08.911660 23083 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:08.900449 23402 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:08.916463 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:08.893646 23377 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:59:08.925763 21175 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:08.906456 23279 consensus_meta.cc:287] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.509s user 0.008s sys 0.004s
I20250831 13:59:08.958492 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1709) took 1445 ms. Trace:
I20250831 13:59:08.959177 20617 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786
I20250831 13:59:08.959067 20502 rpcz_store.cc:276] 0831 13:59:07.513457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.536721 (+1023264us) service_pool.cc:225] Handling call
0831 13:59:08.540704 (+ 3983us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:08.563120 (+ 22416us) raft_consensus.cc:1532] Early marking committed up to index 4
0831 13:59:08.563476 (+ 356us) log.cc:844] Serialized 14 byte log entry
0831 13:59:08.883051 (+319575us) raft_consensus.cc:1537] Triggering prepare for 0 ops
0831 13:59:08.883106 (+ 55us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:08.883148 (+ 42us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.958161 (+ 75013us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b540009a1d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:08.958240 (+ 79us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:08.958417 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31647,"prepare.queue_time_us":893,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":1093,"spinlock_wait_cycles":50950016,"thread_start_us":287516,"threads_started":1}
I20250831 13:59:08.899031 23387 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:08.902887 23378 raft_consensus.cc:513] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:08.965020 20616 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:08.950136 23381 consensus_queue.cc:1230] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20250831 13:59:08.976274 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:08.971343 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 166846us
W20250831 13:59:08.972219 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 164179us
I20250831 13:59:08.951831 23000 raft_consensus.cc:695] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:08.987387 20543 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:08.987206 23400 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:08.990445 23267 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:59:08.999933 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:08.995529 23446 log.cc:927] Time spent T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
W20250831 13:59:09.012001 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 66322.3us
I20250831 13:59:09.012398 20569 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1656) took 1531 ms. Trace:
I20250831 13:59:09.012964 20543 leader_election.cc:304] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:09.013191 23264 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
W20250831 13:59:09.014804 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.015072 23459 raft_consensus.cc:2762] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20250831 13:59:08.995138 23400 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:09.015836 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.017031 23400 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:08.985996 23386 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:59:09.014829 20543 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:09.019380 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 58440.4us
I20250831 13:59:09.020102 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.006336 20502 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 8.
I20250831 13:59:09.007404 23381 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20250831 13:59:09.007846 23164 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.030405 23164 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.032832 23164 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:09.033133 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 86997.8us
I20250831 13:59:09.034116 20617 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.008268 23463 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.009263 23392 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.035061 23400 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:09.035156 23463 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.035823 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.036449 23392 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.023257 20576 raft_consensus.cc:1273] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:09.012758 20569 rpcz_store.cc:276] 0831 13:59:07.481159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:07.481528 (+ 369us) service_pool.cc:225] Handling call
0831 13:59:07.876478 (+394950us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:07.908431 (+ 31953us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:07.908468 (+ 37us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:07.914117 (+ 5649us) log.cc:844] Serialized 26 byte log entry
0831 13:59:08.256779 (+342662us) spinlock_profiling.cc:243] Waited 7.72 ms on lock 0x7b18003d1c20. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a13fb2 00007fb9bc5b42bb 00007fb9bc5b9dc3 00007fb9bc5ba171 00007fb9bc94909f
0831 13:59:08.407339 (+150560us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:08.488637 (+ 81298us) log.cc:844] Serialized 14 byte log entry
0831 13:59:08.489832 (+ 1195us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.490119 (+ 287us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:09.011976 (+521857us) raft_consensus.cc:1683] finished
0831 13:59:09.012088 (+ 112us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:09.012240 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4320,"prepare.queue_time_us":567,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":33139,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":22926,"spinlock_wait_cycles":17746816,"thread_start_us":138269,"threads_started":1,"wal-append.queue_time_us":858470}
I20250831 13:59:09.024971 23459 raft_consensus.cc:2747] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:09.037359 23463 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.026021 20543 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.036732 23400 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:09.037983 23463 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.038173 23400 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.038341 22804 raft_consensus.cc:2802] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:09.038635 23400 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.038944 22804 raft_consensus.cc:695] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:09.043511 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.042048 23459 raft_consensus.cc:2747] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:59:09.052423 20650 raft_consensus.cc:2466] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:09.053918 20649 raft_consensus.cc:1238] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [term 10 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 9. Current term is 10. Ops: []
I20250831 13:59:09.010558 23083 raft_consensus.cc:513] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.050973 23386 raft_consensus.cc:695] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:09.063392 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.069161 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:09.071750 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 45876.4us
I20250831 13:59:09.072553 20642 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
W20250831 13:59:09.073378 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.079568 20650 raft_consensus.cc:1273] T 1e3ed7784853456bbe218b149399b557 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Refusing update from remote peer 212b66976f874aa682cae0fb77928786: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20250831 13:59:09.083559 22804 consensus_queue.cc:237] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.084674 23377 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.052248 23463 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.094432 20576 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 5.
I20250831 13:59:09.098939 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:09.100934 21175 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:09.101435 20501 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 6.
I20250831 13:59:09.078451 23315 raft_consensus.cc:513] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:59:09.075198 23000 consensus_queue.cc:237] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.100028 23381 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.114179 23264 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.089605 23459 consensus_queue.cc:1035] T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20250831 13:59:09.116261 23083 leader_election.cc:290] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.115566 23377 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.116039 22873 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:59:09.170167 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 79514us
I20250831 13:59:09.134624 23381 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:09.169108 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 605264us
W20250831 13:59:09.191360 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.198024 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.204509 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:09.206943 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 151622us
I20250831 13:59:09.207959 20543 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:09.204710 20495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.012s sys 0.010s
W20250831 13:59:09.209817 20495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.012s sys 0.010s
I20250831 13:59:09.152892 23386 consensus_queue.cc:237] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.213843 20647 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
W20250831 13:59:09.215884 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 139225us
I20250831 13:59:09.217825 20649 raft_consensus.cc:1238] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:09.219000 20498 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Rejecting Update request from peer ab7ade51fa134a1b814fab748f01d43d for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:09.222949 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.223618 23436 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.222944 23459 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:09.251616 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:09.252403 20499 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:09.253538 20467 leader_election.cc:304] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:09.254503 23462 raft_consensus.cc:2747] T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:09.255143 22873 leader_election.cc:290] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.256649 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:09.257066 20467 leader_election.cc:304] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:09.257324 20576 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.257916 23264 raft_consensus.cc:2747] T b2d4d2e85a054797ae54ac949008c1f1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 13:59:09.259310 23460 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20250831 13:59:09.259366 20467 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:09.260648 20494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1690) took 2492 ms. Trace:
I20250831 13:59:09.261408 23264 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:59:09.261998 23456 log.cc:927] Time spent T 712d44ed33ec4e37b2bef30d18b3495b P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20250831 13:59:09.261019 20494 rpcz_store.cc:276] 0831 13:59:06.767770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.394248 (+1626478us) service_pool.cc:225] Handling call
0831 13:59:08.562610 (+168362us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:59:08.562865 (+ 255us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:08.563375 (+ 510us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:08.563405 (+ 30us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:08.565632 (+ 2227us) log.cc:844] Serialized 26 byte log entry
0831 13:59:08.977724 (+412092us) raft_consensus.cc:1643] Marking committed up to 3
0831 13:59:08.977778 (+ 54us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:08.977867 (+ 89us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:08.979532 (+ 1665us) spinlock_profiling.cc:243] Waited 4.64 ms on lock 0x7b4800080058. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c1838c41 00007fb9c182fde7 00007fb9c182fc7a 00007fb9c192506f 00007fb9c1924d37 00007fb9c193935c 00007fb9c1939308 00007fb9c1939288 00007fb9c1939243 00007fb9c19384ec 00007fb9c08b6656
0831 13:59:09.260458 (+280926us) raft_consensus.cc:1683] finished
0831 13:59:09.260478 (+ 20us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:09.260594 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6208,"raft.run_cpu_time_us":3622,"raft.run_wall_time_us":408242,"spinlock_wait_cycles":13193856,"thread_start_us":205283,"threads_started":1,"wal-append.queue_time_us":545861}
I20250831 13:59:09.263445 23400 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.265493 20576 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 9.
W20250831 13:59:09.269335 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.266580 20467 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
W20250831 13:59:09.269758 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.270534 23264 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:59:09.270987 23264 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:09.271567 23264 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:09.271711 21175 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters:
W20250831 13:59:09.273152 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 70339.2us
I20250831 13:59:09.273943 20543 leader_election.cc:304] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:09.270933 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:09.275122 20648 raft_consensus.cc:2391] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:09.275151 20543 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.276008 23190 raft_consensus.cc:2747] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:09.276618 23197 raft_consensus.cc:2747] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:59:09.277568 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:09.278605 22873 raft_consensus.cc:2802] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:09.280761 23264 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.280916 20543 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.285871 23378 leader_election.cc:290] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.287451 23159 raft_consensus.cc:3058] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:09.282850 23264 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:09.319798 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.283684 20467 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:59:09.292634 23461 log.cc:927] Time spent T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20250831 13:59:09.285670 23404 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.303128 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:09.303570 23466 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:09.249130 23315 leader_election.cc:290] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131)
W20250831 13:59:09.338527 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 62088.4us
I20250831 13:59:09.306139 23463 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.306318 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:09.290664 23453 log.cc:927] Time spent T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.324s user 0.000s sys 0.002s
I20250831 13:59:09.363729 23400 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:09.329336 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:09.364285 23470 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.099s user 0.000s sys 0.006s
I20250831 13:59:09.364809 20575 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:09.326433 20466 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.230854 23457 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:09.366232 23457 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.367522 20466 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.368543 23457 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.369457 22819 consensus_queue.cc:1046] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:09.369590 23308 consensus_queue.cc:1046] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:09.370270 22819 consensus_queue.cc:1230] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250831 13:59:09.370682 20466 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.371088 23308 raft_consensus.cc:2762] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20250831 13:59:09.371451 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.330282 23190 raft_consensus.cc:2747] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:09.334504 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1688) took 2567 ms. Trace:
I20250831 13:59:09.335554 23264 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:59:09.336447 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.332811 23466 raft_consensus.cc:491] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.305133 23381 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.355041 20651 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.223073 23370 consensus_queue.cc:260] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.372462 20495 rpcz_store.cc:276] 0831 13:59:06.767423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:07.234327 (+466904us) service_pool.cc:225] Handling call
0831 13:59:07.234638 (+ 311us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:07.234994 (+ 356us) raft_consensus.cc:1532] Early marking committed up to index 5
0831 13:59:08.258544 (+1023550us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b1800504b80. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9c26b9e31 00007fb9c26c6781 00007fb9b99f70f7 00007fb9c2632b89 00007fb9b9a13a4b 00007fb9b9a11cef 00007fb9b9a11805 00007fb9c035b327 00007fb9bc980957 00007fb9bc99006f 00007fb9bc98ffe8
0831 13:59:08.404489 (+145945us) log.cc:844] Serialized 14 byte log entry
0831 13:59:09.227816 (+823327us) log.cc:844] Serialized 14 byte log entry
0831 13:59:09.241834 (+ 14018us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:09.279855 (+ 38021us) log.cc:844] Serialized 26 byte log entry
0831 13:59:09.280010 (+ 155us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:09.280040 (+ 30us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:09.280131 (+ 91us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:09.334219 (+ 54088us) raft_consensus.cc:1683] finished
0831 13:59:09.334245 (+ 26us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:09.334420 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9406,"prepare.queue_time_us":652823,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":99,"raft.queue_time_us":18246,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":7898,"spinlock_wait_cycles":6707584,"thread_start_us":794628,"threads_started":2,"wal-append.queue_time_us":678194}
I20250831 13:59:09.338744 23436 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.368628 23462 raft_consensus.cc:2802] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 13:59:09.336838 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 53344.8us
I20250831 13:59:09.378026 20651 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:09.360045 23402 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.379627 21175 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:09.384240 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:09.385478 23436 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.385843 23462 raft_consensus.cc:491] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.386518 20467 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:09.387277 20574 raft_consensus.cc:1273] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20250831 13:59:09.395344 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:09.403534 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.404812 23380 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:09.405351 23380 raft_consensus.cc:491] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.405557 20543 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.406481 23190 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.406559 23308 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 13:59:09.406973 20543 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.377171 23466 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:09.405738 23380 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.396838 23377 raft_consensus.cc:3053] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:09.383251 22819 raft_consensus.cc:3053] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 9 LEADER]: Stepping down as leader of term 9
I20250831 13:59:09.423847 20467 leader_election.cc:304] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:09.398414 23463 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:09.398940 23462 raft_consensus.cc:3058] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.369483 23404 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.397111 23245 consensus_queue.cc:1046] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20250831 13:59:09.441273 20642 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.425092 23377 raft_consensus.cc:738] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:09.427495 23463 raft_consensus.cc:491] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.426996 22819 raft_consensus.cc:738] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:09.447331 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0c14599239fd4c99b3331692a513a092" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:09.450274 20495 raft_consensus.cc:3058] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:59:09.424760 23435 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.015s sys 0.030s
I20250831 13:59:09.435345 23159 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:09.470049 20543 leader_election.cc:400] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4.
I20250831 13:59:09.470476 20543 leader_election.cc:403] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20250831 13:59:09.451956 23463 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.450071 23264 raft_consensus.cc:2747] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:59:09.483705 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:09.466461 23435 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.724s user 0.016s sys 0.030s
I20250831 13:59:09.472306 23245 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:09.487219 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:09.490303 21175 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:09.487295 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 259790us
W20250831 13:59:09.488917 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 81480.1us
I20250831 13:59:09.486812 23381 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 3. Current term is 4.
I20250831 13:59:09.505854 23474 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.508364 20616 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.497565 23467 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.493686 23245 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.518884 20467 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:09.521371 20543 leader_election.cc:304] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.498701 23377 consensus_queue.cc:260] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.523687 23377 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.498828 22819 consensus_queue.cc:260] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.529582 22819 raft_consensus.cc:3058] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:09.506423 23474 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.548468 20643 raft_consensus.cc:1238] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:09.536620 23387 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 13:59:09.559793 23463 raft_consensus.cc:513] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.564669 23404 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.547680 23381 raft_consensus.cc:2747] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 13:59:09.544659 22873 raft_consensus.cc:695] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [term 9 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:59:09.576771 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.579164 20495 raft_consensus.cc:2466] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 9.
W20250831 13:59:09.578500 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 63280.7us
I20250831 13:59:09.521121 23467 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:09.585115 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 80922.2us
W20250831 13:59:09.585853 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 82582.9us
I20250831 13:59:09.592303 23474 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.599186 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:59:09.599480 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.599845 20495 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.610152 21206 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
W20250831 13:59:09.611827 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 224573us
I20250831 13:59:09.613373 20498 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37606 (request call id 90) took 2811 ms. Trace:
I20250831 13:59:09.613860 20498 rpcz_store.cc:276] 0831 13:59:06.801955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.392179 (+1590224us) service_pool.cc:225] Handling call
0831 13:59:08.489096 (+ 96917us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b5800000f98. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9b9857c41 00007fb9b99d7251 00007fb9b99d7102 00007fb9c26d2c39 00007fb9c185e000 00007fb9c1924eab 00007fb9c1924f31 00007fb9c18e7f50 00007fb9c18d3199 00007fb9c18d2c42 00007fb9baf5fb44
0831 13:59:08.519910 (+ 30814us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:09.219456 (+699546us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:09.587413 (+367957us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b54001e0854. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:09.613109 (+ 25696us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b54001e0850. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97586d 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:09.613283 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1667565312}
I20250831 13:59:09.599522 23467 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.644343 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:09.645072 20575 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.648118 23259 raft_consensus.cc:3053] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:09.648666 23259 raft_consensus.cc:738] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:09.649732 23259 consensus_queue.cc:260] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.628098 23159 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 13:59:09.629159 20574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1667) took 1159 ms. Trace:
I20250831 13:59:09.663336 20574 rpcz_store.cc:276] 0831 13:59:08.470166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:08.521021 (+ 50855us) service_pool.cc:225] Handling call
0831 13:59:08.525291 (+ 4270us) raft_consensus.cc:1469] Updating replica for 0 ops
0831 13:59:09.396692 (+871401us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:09.628937 (+232245us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b540031ddd4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc976dc1 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32 00007fb9bad75ccc 00007fb9bad75106
0831 13:59:09.629100 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1981021440}
I20250831 13:59:09.630793 20467 leader_election.cc:304] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:09.631856 23458 consensus_queue.cc:1035] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:09.603551 23245 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:09.635782 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 135750us
I20250831 13:59:09.636698 20643 raft_consensus.cc:1238] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 4. Current term is 5. Ops: []
I20250831 13:59:09.636610 23392 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:09.637452 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.638209 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.668648 20645 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:09.638641 20644 raft_consensus.cc:1238] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:09.639281 20650 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20250831 13:59:09.639431 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 51026.9us
I20250831 13:59:09.639597 23000 consensus_queue.cc:1035] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:59:09.668260 20647 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 5.
I20250831 13:59:09.640818 23279 raft_consensus.cc:3053] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Stepping down as leader of term 4
I20250831 13:59:09.640836 23435 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.642154 23370 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:09.642967 23463 leader_election.cc:290] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.643563 22873 consensus_queue.cc:237] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.678885 20466 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.679739 20466 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.680470 23267 consensus_queue.cc:1046] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:59:09.652269 23462 raft_consensus.cc:513] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.681061 23468 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:09.681470 23267 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250831 13:59:09.682049 23267 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.681527 23468 raft_consensus.cc:491] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.682423 23267 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:09.682654 23462 leader_election.cc:290] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.683290 23462 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.683707 23462 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.684556 23435 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.685535 23462 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:09.686036 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.652649 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "90a0a56228d04660b3cb351485be6593" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:09.687150 20649 raft_consensus.cc:3058] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.652472 21174 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:59:09.624200 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.668243 23467 raft_consensus.cc:2802] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20250831 13:59:09.691042 20543 connection.cc:232] RPC call timeout handler was delayed by -1.2616659000000254s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18810000000000002s for a response
I20250831 13:59:09.691372 23467 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:59:09.691671 20543 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.161s (SENT) [suppressed 41 similar messages]
I20250831 13:59:09.691888 23467 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:09.654891 23466 raft_consensus.cc:513] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.691988 23245 consensus_queue.cc:1046] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 13:59:09.692966 23264 raft_consensus.cc:3053] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Stepping down as leader of term 7
I20250831 13:59:09.693290 23264 raft_consensus.cc:738] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:09.693815 23466 leader_election.cc:290] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.639966 23436 consensus_queue.cc:1046] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250831 13:59:09.693990 23190 raft_consensus.cc:2747] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:09.694049 23264 consensus_queue.cc:260] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.668713 23392 raft_consensus.cc:491] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:09.695340 23392 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.695719 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:09.696301 20574 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:59:09.696934 23264 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
W20250831 13:59:09.677327 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 46650.2us
I20250831 13:59:09.671583 23279 raft_consensus.cc:738] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:09.700817 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:09.701927 20495 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 8.
I20250831 13:59:09.703092 23000 consensus_queue.cc:1035] T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20250831 13:59:09.703745 23436 consensus_queue.cc:1230] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250831 13:59:09.694861 23463 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250831 13:59:09.705142 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.705834 20647 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:09.706228 23386 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:59:09.703464 23279 consensus_queue.cc:260] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.699750 23435 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.705789 20467 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:09.706625 20616 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.699158 23259 raft_consensus.cc:3058] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:09.707293 23279 raft_consensus.cc:3058] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Advancing to term 5
I20250831 13:59:09.723371 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.682600 23468 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.725440 20649 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 8.
I20250831 13:59:09.730799 20466 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.734018 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62fa3a4f7ea043fdbcffdcc84f9c28c0" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:09.734903 20495 raft_consensus.cc:3058] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:59:09.737855 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 46211.3us
I20250831 13:59:09.738626 20575 raft_consensus.cc:2466] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 8.
I20250831 13:59:09.737071 20647 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 11.
I20250831 13:59:09.739729 21175 leader_election.cc:304] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:09.703650 23245 consensus_queue.cc:1035] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20250831 13:59:09.741190 20466 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.740409 23436 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20250831 13:59:09.747704 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.746937 23464 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20250831 13:59:09.745522 23435 raft_consensus.cc:2747] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250831 13:59:09.747104 23478 raft_consensus.cc:2802] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
W20250831 13:59:09.755860 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 50162.2us
I20250831 13:59:09.745550 23402 consensus_queue.cc:1046] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.004s
I20250831 13:59:09.751899 20495 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:09.742350 23466 raft_consensus.cc:2802] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:09.759285 23190 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.754580 23436 raft_consensus.cc:491] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.760092 23436 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.759817 23190 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.761933 23190 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:59:09.762784 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.763307 23466 raft_consensus.cc:695] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [term 8 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:59:09.763718 20467 leader_election.cc:304] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:09.764084 23478 raft_consensus.cc:695] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:09.772855 22804 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250831 13:59:09.756467 23464 raft_consensus.cc:491] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.768322 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.780668 20574 raft_consensus.cc:2466] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 7.
I20250831 13:59:09.780853 23083 consensus_queue.cc:1035] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20250831 13:59:09.766932 23487 consensus_queue.cc:1035] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.015s
I20250831 13:59:09.758131 23482 raft_consensus.cc:3053] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:09.780706 23267 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:09.786878 20543 leader_election.cc:336] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.258s (SENT)
I20250831 13:59:09.788089 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:09.797150 20576 raft_consensus.cc:2466] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:09.792555 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:09.793988 20396 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:54880 (request call id 40) took 13676 ms (client timeout 15000 ms). Trace:
I20250831 13:59:09.798369 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:09.798486 20543 leader_election.cc:304] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.796303 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:09.775960 23435 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.801637 23402 raft_consensus.cc:3053] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 11 LEADER]: Stepping down as leader of term 11
I20250831 13:59:09.802103 23402 raft_consensus.cc:738] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:09.802779 23267 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.773010 23478 consensus_queue.cc:237] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.802973 20572 raft_consensus.cc:2466] T 3359826124a94225a4cd8932ecb7fcff P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:09.791282 22819 raft_consensus.cc:2802] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:09.803632 22819 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:09.803825 20569 raft_consensus.cc:2466] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:09.804005 22819 raft_consensus.cc:3058] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.791353 23190 raft_consensus.cc:491] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.784638 23464 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Advancing to term 12
I20250831 13:59:09.825948 20568 raft_consensus.cc:3058] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.826159 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:09.827226 20646 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:09.766961 23392 raft_consensus.cc:513] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.794878 23482 raft_consensus.cc:738] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:09.764899 23466 consensus_queue.cc:237] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.836727 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:09.837496 20569 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:09.830932 23387 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.759312 21176 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:09.797936 23474 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:09.838762 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 75164.1us
I20250831 13:59:09.841135 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:09.811069 23159 raft_consensus.cc:2747] T c719f2a4c3ea4418937337e8e3dd2164 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20250831 13:59:09.849665 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 75128.3us
W20250831 13:59:09.798218 20396 rpcz_store.cc:269] 0831 13:58:56.117850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:58:56.123281 (+ 5431us) service_pool.cc:225] Handling call
0831 13:59:09.793865 (+13670584us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0831 13:59:08.971433 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0831 13:59:08.972344 (+ 911us) write_op.cc:432] Acquiring schema lock in shared mode
0831 13:59:08.972415 (+ 71us) write_op.cc:435] Acquired schema lock
0831 13:59:08.972443 (+ 28us) tablet.cc:660] Decoding operations
0831 13:59:09.023765 (+ 51322us) write_op.cc:620] Acquiring the partition lock for write op
0831 13:59:09.049852 (+ 26087us) write_op.cc:641] Partition lock acquired for write op
0831 13:59:09.050381 (+ 529us) tablet.cc:683] Acquiring locks for 2 operations
0831 13:59:09.077227 (+ 26846us) tablet.cc:699] Row locks acquired
0831 13:59:09.078164 (+ 937us) write_op.cc:260] PREPARE: finished
0831 13:59:09.091954 (+ 13790us) write_op.cc:270] Start()
0831 13:59:09.118275 (+ 26321us) write_op.cc:276] Timestamp: P: 1756648749086875 usec, L: 0
0831 13:59:09.124374 (+ 6099us) op_driver.cc:348] REPLICATION: starting
0831 13:59:09.199970 (+ 75596us) log.cc:844] Serialized 988 byte log entry
0831 13:59:09.698772 (+498802us) op_driver.cc:464] REPLICATION: finished
0831 13:59:09.763290 (+ 64518us) write_op.cc:301] APPLY: starting
0831 13:59:09.763390 (+ 100us) tablet.cc:1365] starting BulkCheckPresence
0831 13:59:09.763519 (+ 129us) tablet.cc:1368] finished BulkCheckPresence
0831 13:59:09.763662 (+ 143us) tablet.cc:1370] starting ApplyRowOperation cycle
0831 13:59:09.764878 (+ 1216us) tablet.cc:1381] finished ApplyRowOperation cycle
0831 13:59:09.765165 (+ 287us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0831 13:59:09.765253 (+ 88us) write_op.cc:312] APPLY: finished
0831 13:59:09.765545 (+ 292us) log.cc:844] Serialized 28 byte log entry
0831 13:59:09.765722 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:59:09.765806 (+ 84us) write_op.cc:454] Released schema lock
0831 13:59:09.765861 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0831 13:59:09.766278 (+ 417us) write_op.cc:489] Releasing partition, row and schema locks
0831 13:59:09.766318 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2615371,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":64201,"apply.run_cpu_time_us":2448,"apply.run_wall_time_us":2938,"num_ops":2,"prepare.queue_time_us":560247,"prepare.run_cpu_time_us":60899,"prepare.run_wall_time_us":457312,"raft.queue_time_us":121831,"raft.run_cpu_time_us":27828,"raft.run_wall_time_us":63009,"replication_time_us":554165,"thread_start_us":646748,"threads_started":4,"wal-append.queue_time_us":336350}]]}
W20250831 13:59:09.865725 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.850517 23482 consensus_queue.cc:260] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.854768 23387 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.855078 22873 raft_consensus.cc:2802] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:09.806905 23404 consensus_queue.cc:1035] T ad68e3a13b9e479791179f35e91bea4b P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20250831 13:59:09.861861 22819 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.861706 20398 catalog_manager.cc:5582] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 7 to 8, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 8 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
W20250831 13:59:09.854980 21176 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 95773.1us
I20250831 13:59:09.799031 23400 consensus_queue.cc:1035] T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250831 13:59:09.829283 23190 raft_consensus.cc:513] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:09.856447 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 69422us
I20250831 13:59:09.858234 23468 raft_consensus.cc:513] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.874527 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:09.823367 23435 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:09.864595 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 91361us
I20250831 13:59:09.877194 20572 raft_consensus.cc:3058] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.881444 20568 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:09.802994 23402 consensus_queue.cc:260] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.870129 23197 raft_consensus.cc:491] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:09.886021 20570 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:09.882057 23487 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.886932 23197 raft_consensus.cc:513] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.887562 23487 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.888991 23197 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.889590 23487 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.875080 22873 raft_consensus.cc:695] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [term 7 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:59:09.889011 23190 leader_election.cc:290] T ad68e3a13b9e479791179f35e91bea4b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.891485 23392 leader_election.cc:290] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:09.891280 22873 consensus_queue.cc:237] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:59:09.892139 23494 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.110s user 0.004s sys 0.000s
I20250831 13:59:09.892927 20467 leader_election.cc:304] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:09.894425 20467 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:09.895758 23377 raft_consensus.cc:2747] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250831 13:59:09.896517 23377 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:09.883929 20398 catalog_manager.cc:5582] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 4 to 9, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 9 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:59:09.903501 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:09.904755 20572 raft_consensus.cc:2466] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:09.910435 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:09.910559 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:09.911360 20649 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.885938 23486 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.914531 20646 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:09.896917 23377 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:09.917629 21208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:09.873744 23474 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.883725 23482 raft_consensus.cc:3058] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:09.916325 20568 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:09.916754 23402 raft_consensus.cc:3058] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Advancing to term 12
W20250831 13:59:09.916879 20543 leader_election.cc:336] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.600s (SENT)
I20250831 13:59:09.917119 20466 leader_election.cc:304] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:09.920033 23486 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:09.922106 23486 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.922860 23259 raft_consensus.cc:2802] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader election won for term 6
I20250831 13:59:09.923987 21208 raft_consensus.cc:2408] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20250831 13:59:09.918373 20570 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:59:09.925644 20543 leader_election.cc:304] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:09.896912 23468 leader_election.cc:290] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.924460 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:09.923302 23377 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:09.929320 23259 raft_consensus.cc:695] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [term 6 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:09.929977 20495 raft_consensus.cc:1273] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20250831 13:59:09.930225 23259 consensus_queue.cc:237] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.931869 22819 leader_election.cc:290] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:09.943701 20569 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:09.882894 23380 raft_consensus.cc:513] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.947130 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:09.949028 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5806109eefc8457d8e9110b9ec030873" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:09.949613 20502 raft_consensus.cc:3058] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:09.952641 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:09.955612 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:09.955782 21205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:09.956409 20493 raft_consensus.cc:2391] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 9: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:09.958853 20649 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:09.959753 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:09.960847 20501 raft_consensus.cc:2408] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 13:59:09.961366 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:09.962930 20498 raft_consensus.cc:1238] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 9. Current term is 10. Ops: []
I20250831 13:59:09.940073 23381 raft_consensus.cc:2747] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:09.965117 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:09.965694 20398 catalog_manager.cc:5582] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 4 to 5, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 5 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
I20250831 13:59:09.967232 20466 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:09.968379 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 74038.3us
I20250831 13:59:09.926422 23436 raft_consensus.cc:513] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.926316 23467 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.970538 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:09.971485 20495 raft_consensus.cc:2408] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20250831 13:59:09.972215 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:09.972536 20568 raft_consensus.cc:1273] T e93846478efb47a1b008fc73d5d25914 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:09.973223 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:09.974107 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:09.971766 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:09.975181 20493 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:09.976270 21209 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d"
I20250831 13:59:09.976796 21209 raft_consensus.cc:3058] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:09.977114 23436 leader_election.cc:290] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:09.977811 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 60998.5us
W20250831 13:59:09.978485 20543 leader_election.cc:336] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.631s (SENT)
I20250831 13:59:09.978854 20543 leader_election.cc:304] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:09.979666 20499 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:09.971819 23468 raft_consensus.cc:2802] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:09.982743 23475 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:09.983186 23468 raft_consensus.cc:695] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [term 9 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:09.983336 23475 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.984072 23468 consensus_queue.cc:237] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:09.985319 23475 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:10.001180 21209 raft_consensus.cc:2466] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:10.022594 20467 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:10.027443 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:10.028268 20650 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:10.044935 20501 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:10.014900 22873 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:10.063941 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 85492us
I20250831 13:59:10.042296 23381 raft_consensus.cc:2747] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:59:10.077402 20398 catalog_manager.cc:5582] T 5ce6782647ef46f095f726e552f5ad23 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 3 to 4, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 4 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:59:10.051409 23000 raft_consensus.cc:491] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:10.096629 20543 leader_election.cc:336] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.688s (SENT)
I20250831 13:59:10.068953 23279 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20250831 13:59:10.103340 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 99936.5us
I20250831 13:59:10.104089 20575 raft_consensus.cc:1273] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:59:10.107839 20501 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: []
I20250831 13:59:10.102614 20398 catalog_manager.cc:5582] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 6, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 6 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250831 13:59:10.100668 23435 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:10.129704 20398 catalog_manager.cc:5582] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 5 to 8, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 8 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:59:10.133335 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.133971 20501 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
W20250831 13:59:10.134080 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:10.077020 22873 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:59:10.137748 20468 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:10.122570 21205 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
W20250831 13:59:10.142949 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 56442.9us
I20250831 13:59:10.147188 20398 catalog_manager.cc:5582] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:59:10.150017 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:10.128676 23474 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:10.149655 21205 pending_rounds.cc:85] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Aborting all ops after (but not including) 8
W20250831 13:59:10.201146 23500 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.124s user 0.001s sys 0.001s
W20250831 13:59:10.202888 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:10.205832 20498 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250831 13:59:10.206588 20398 catalog_manager.cc:5582] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 3 to 6, leader changed from 44d163094add49049cc749547d891602 (127.18.233.131) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 6 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:59:10.213985 23387 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:10.214152 22873 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:10.209398 23501 log.cc:927] Time spent T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.191s user 0.001s sys 0.000s
W20250831 13:59:10.210456 20543 leader_election.cc:336] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.718s (SENT)
I20250831 13:59:10.215574 20543 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.216307 23159 raft_consensus.cc:2747] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:59:10.217332 20543 leader_election.cc:336] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.738s (SENT)
I20250831 13:59:10.217694 20543 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.218396 23159 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20250831 13:59:10.211796 23503 log.cc:927] Time spent T f29595dd2c044618bd6cb4517fbc5c7b P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.194s user 0.002s sys 0.000s
I20250831 13:59:10.213585 20398 catalog_manager.cc:5582] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 2 to 5, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129). New cstate: current_term: 5 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: HEALTHY } } }
I20250831 13:59:10.220005 22873 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 13:59:10.220642 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.202284 23377 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.221480 20498 raft_consensus.cc:2408] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ab7ade51fa134a1b814fab748f01d43d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20250831 13:59:10.218850 20543 leader_election.cc:336] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.780s (SENT)
I20250831 13:59:10.119041 23000 raft_consensus.cc:513] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.229388 20502 raft_consensus.cc:2466] T 5806109eefc8457d8e9110b9ec030873 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:10.149324 23380 leader_election.cc:290] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:10.197161 21205 pending_rounds.cc:107] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20250831 13:59:10.197903 23467 leader_election.cc:290] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 6 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:10.277238 20617 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d; no voters:
I20250831 13:59:10.270174 21205 raft_consensus.cc:2887] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20250831 13:59:10.288219 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 69439.5us
I20250831 13:59:10.272156 23083 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:10.279096 23498 log.cc:927] Time spent T 1e3ed7784853456bbe218b149399b557 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.349s user 0.000s sys 0.001s
I20250831 13:59:10.291301 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.292752 20398 catalog_manager.cc:5582] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 5 to 6, leader changed from 212b66976f874aa682cae0fb77928786 (127.18.233.130) to 44d163094add49049cc749547d891602 (127.18.233.131). New cstate: current_term: 6 leader_uuid: "44d163094add49049cc749547d891602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } }
W20250831 13:59:10.295209 23510 log.cc:927] Time spent T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.099s user 0.000s sys 0.000s
I20250831 13:59:10.281534 23487 raft_consensus.cc:2802] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election won for term 8
W20250831 13:59:10.297024 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:10.295115 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:10.299772 20543 leader_election.cc:336] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.798s (SENT)
I20250831 13:59:10.303225 20543 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.303447 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.306095 20498 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:10.307248 20496 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:10.303233 20497 raft_consensus.cc:1238] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 5. Current term is 6. Ops: []
I20250831 13:59:10.306520 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "995d9ef0c84a4aae8581ab92c0607fdf" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.308364 20575 raft_consensus.cc:3058] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:10.309649 20398 catalog_manager.cc:5582] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "58aded50ab0d4609a053647eb11b1f48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } health_report { overall_health: UNKNOWN } } }
I20250831 13:59:10.304563 20617 leader_election.cc:304] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:10.321838 20571 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
I20250831 13:59:10.293623 23083 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:10.344491 23508 log.cc:927] Time spent T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20250831 13:59:10.345582 23083 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:10.346575 20496 raft_consensus.cc:1273] T e93846478efb47a1b008fc73d5d25914 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20250831 13:59:10.351538 23377 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:10.340754 23435 consensus_queue.cc:1046] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20250831 13:59:10.352806 23435 raft_consensus.cc:3053] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:10.353164 23435 raft_consensus.cc:738] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:59:10.355083 23504 log.cc:927] Time spent T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.245s user 0.001s sys 0.000s
I20250831 13:59:10.355376 23435 consensus_queue.cc:260] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.356634 23435 raft_consensus.cc:3058] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:10.303882 23400 consensus_queue.cc:1035] T a7956988b57844cfad1028024ac745e6 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20250831 13:59:10.358753 23467 consensus_queue.cc:1035] T 1c85d8618b084c51b4e509df8650069d P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20250831 13:59:10.358846 23387 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20250831 13:59:10.360803 20569 raft_consensus.cc:1273] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20250831 13:59:10.361722 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.362326 20569 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:10.363008 23520 raft_consensus.cc:2747] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:10.363420 20616 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:10.363653 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.363225 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.364245 20616 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:10.364279 20570 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:10.426173 20575 raft_consensus.cc:2466] T 995d9ef0c84a4aae8581ab92c0607fdf P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:10.428299 20569 raft_consensus.cc:2391] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
W20250831 13:59:10.430385 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:10.433769 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 43928.3us
I20250831 13:59:10.437510 20650 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:10.320021 23487 raft_consensus.cc:695] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:10.439217 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8de1861ba5b5442381501e5b214a51f9" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.443285 20569 raft_consensus.cc:2391] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:10.439828 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.445757 20650 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 9. Current term is 10. Ops: []
I20250831 13:59:10.449074 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:10.449390 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 85211.5us
I20250831 13:59:10.449741 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.450737 20570 raft_consensus.cc:2391] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:10.451413 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.454013 20569 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:10.455014 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.455651 20569 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 9.
I20250831 13:59:10.457692 21175 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:10.428253 23513 raft_consensus.cc:2802] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:10.460557 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:10.364593 23502 raft_consensus.cc:2747] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250831 13:59:10.466624 20616 leader_election.cc:304] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:10.465525 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.482136 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.484189 20616 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:10.481587 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.491950 23400 raft_consensus.cc:2802] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:10.492604 23400 raft_consensus.cc:695] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:59:10.492758 23515 log.cc:927] Time spent T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20250831 13:59:10.493592 23400 consensus_queue.cc:237] T 501e1c372209405c80034fe2411aaa22 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:10.494323 23521 log.cc:927] Time spent T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20250831 13:59:10.449178 23190 raft_consensus.cc:491] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:10.450614 23381 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:10.499703 20616 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.500826 20575 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 5.
I20250831 13:59:10.463724 23083 raft_consensus.cc:2802] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:10.462779 23513 raft_consensus.cc:491] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:10.514034 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 65038.5us
I20250831 13:59:10.515177 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:10.516546 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:10.521512 20572 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:10.524006 20570 raft_consensus.cc:1238] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:10.499897 23392 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:10.502175 23466 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:59:10.540267 20616 leader_election.cc:304] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:10.540521 20496 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250831 13:59:10.512758 23000 leader_election.cc:290] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:10.519711 23478 raft_consensus.cc:2802] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:10.517520 23083 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:10.545650 20570 raft_consensus.cc:1273] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20250831 13:59:10.545948 20571 raft_consensus.cc:1273] T 0c14599239fd4c99b3331692a513a092 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20250831 13:59:10.519131 23474 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:59:10.543287 23466 raft_consensus.cc:491] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:10.520601 23513 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:10.527069 23392 raft_consensus.cc:491] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250831 13:59:10.566776 23506 log.cc:927] Time spent T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.288s user 0.000s sys 0.001s
I20250831 13:59:10.566877 23392 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:10.539355 23499 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:10.571916 23499 raft_consensus.cc:491] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:10.572371 23499 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:10.558900 23487 consensus_queue.cc:237] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:10.518385 23381 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:10.583150 23381 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
I20250831 13:59:10.548182 22804 raft_consensus.cc:2802] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader election won for term 10
I20250831 13:59:10.512637 23190 raft_consensus.cc:513] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:10.559027 20543 leader_election.cc:336] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.946s (SENT)
I20250831 13:59:10.512359 23502 consensus_queue.cc:1046] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20250831 13:59:10.559376 23478 raft_consensus.cc:695] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:10.563453 23466 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 9 FOLLOWER]: Advancing to term 10
W20250831 13:59:10.558404 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 96953.8us
I20250831 13:59:10.558820 23474 raft_consensus.cc:491] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:10.593953 20543 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.557664 23083 raft_consensus.cc:3058] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:10.599455 23502 raft_consensus.cc:3053] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 9 LEADER]: Stepping down as leader of term 9
I20250831 13:59:10.555893 23520 consensus_queue.cc:1035] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.043s
I20250831 13:59:10.606377 23502 raft_consensus.cc:738] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 212b66976f874aa682cae0fb77928786, State: Running, Role: LEADER
I20250831 13:59:10.600140 22804 raft_consensus.cc:695] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [term 10 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:10.607543 23502 consensus_queue.cc:260] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.608052 22804 consensus_queue.cc:237] T 5806109eefc8457d8e9110b9ec030873 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.609148 23502 raft_consensus.cc:3058] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:10.616073 20571 raft_consensus.cc:1238] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:10.615461 23478 consensus_queue.cc:237] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } }
I20250831 13:59:10.617082 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.617679 20571 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
W20250831 13:59:10.619720 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 69908.9us
I20250831 13:59:10.619966 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:10.620419 20543 leader_election.cc:336] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.985s (SENT)
I20250831 13:59:10.620576 20570 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 7
W20250831 13:59:10.621346 20543 leader_election.cc:336] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.999s (SENT)
I20250831 13:59:10.622113 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.623292 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "62d7feaf5302450caacdf561ce6f06d8" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.623546 20568 raft_consensus.cc:1238] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:10.623987 20576 raft_consensus.cc:2391] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
W20250831 13:59:10.624280 23482 consensus_meta.cc:287] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.697s user 0.008s sys 0.000s
I20250831 13:59:10.624459 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.624692 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.625929 20495 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20250831 13:59:10.622277 20543 leader_election.cc:336] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.093s (SENT)
I20250831 13:59:10.618495 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f10d72d324374615bbfb06fd2c0e548d" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.626987 20543 leader_election.cc:304] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.633833 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.636484 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:59:10.637918 23525 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20250831 13:59:10.637831 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:10.638229 23190 leader_election.cc:290] T 79dc07c2112a46fe82d7250aa95bd6dc P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:10.604077 23474 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:10.604297 23475 consensus_queue.cc:1035] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20250831 13:59:10.644734 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:10.645750 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.647256 23000 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:10.647688 23000 raft_consensus.cc:491] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:10.648108 23000 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:10.647794 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.648916 20571 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 4.
I20250831 13:59:10.631083 23381 raft_consensus.cc:2747] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:10.653038 23466 raft_consensus.cc:513] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:10.655596 23499 raft_consensus.cc:513] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.656613 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.657203 20568 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:10.657919 23499 leader_election.cc:290] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:10.657994 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.658823 20568 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:10.659603 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.660817 20568 raft_consensus.cc:1273] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20250831 13:59:10.665436 23523 log.cc:927] Time spent T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.103s user 0.000s sys 0.000s
I20250831 13:59:10.666443 23475 raft_consensus.cc:491] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:10.667119 23475 raft_consensus.cc:513] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.669456 23475 leader_election.cc:290] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:10.670572 20570 raft_consensus.cc:2466] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:10.671420 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.672067 20571 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 10.
W20250831 13:59:10.675446 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 53210.2us
I20250831 13:59:10.676919 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:10.677134 20543 leader_election.cc:336] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.841s (SENT)
I20250831 13:59:10.677568 20543 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.677652 23466 leader_election.cc:290] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 10 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:10.677572 20571 raft_consensus.cc:3058] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Advancing to term 6
I20250831 13:59:10.678849 23381 raft_consensus.cc:2747] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:10.679543 23381 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:10.680861 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.681602 20575 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:10.683916 23392 raft_consensus.cc:513] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.685168 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.685688 20570 raft_consensus.cc:3058] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:10.685817 23392 leader_election.cc:290] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:10.687824 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 262245us
I20250831 13:59:10.690729 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346d047c73c648a5aab2f44efa73aee7" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:10.696312 20466 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:10.698458 23387 raft_consensus.cc:2802] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:10.699080 23387 raft_consensus.cc:695] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:10.699582 23513 raft_consensus.cc:513] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.700138 23387 consensus_queue.cc:237] T 7f52ce7625424dcc943a573347b456bb P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.701680 23513 leader_election.cc:290] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:10.706107 23505 raft_consensus.cc:491] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:10.677762 23517 log.cc:927] Time spent T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20250831 13:59:10.706645 23505 raft_consensus.cc:513] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:10.712420 23464 consensus_meta.cc:287] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.846s user 0.006s sys 0.003s
I20250831 13:59:10.712965 20571 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 6.
I20250831 13:59:10.712806 23464 raft_consensus.cc:513] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.713367 23505 leader_election.cc:290] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:10.706881 23402 consensus_queue.cc:1046] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.008s
I20250831 13:59:10.717128 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:10.682605 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.720477 20575 raft_consensus.cc:3058] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:10.726583 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:10.727319 20649 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:10.730046 20543 leader_election.cc:304] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
W20250831 13:59:10.732970 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 54297us
I20250831 13:59:10.733803 20543 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:10.748776 23381 raft_consensus.cc:2747] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 13:59:10.774389 20647 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250831 13:59:10.775326 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:10.775964 20643 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:10.776304 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:10.778425 23190 raft_consensus.cc:2747] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:10.779488 23464 leader_election.cc:290] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 election: Requested vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:10.734615 23474 raft_consensus.cc:513] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.774075 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:10.781354 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 57924.4us
W20250831 13:59:10.782804 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 67276.1us
I20250831 13:59:10.783597 20576 raft_consensus.cc:1273] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20250831 13:59:10.783941 20576 pending_rounds.cc:85] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Aborting all ops after (but not including) 4
I20250831 13:59:10.784127 20576 pending_rounds.cc:107] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20250831 13:59:10.784317 20576 raft_consensus.cc:2887] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:10.785800 20617 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:10.786515 20571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.787029 20617 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:10.787336 20571 raft_consensus.cc:3058] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:59:10.790802 20570 raft_consensus.cc:2466] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:10.793243 20575 raft_consensus.cc:2466] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
I20250831 13:59:10.794242 20616 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:10.801249 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:10.813506 20651 raft_consensus.cc:1273] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20250831 13:59:10.818202 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:10.818835 20649 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:10.821327 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.822046 20574 raft_consensus.cc:3058] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:10.823947 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:10.824563 20576 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:10.824712 20646 raft_consensus.cc:2391] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 10: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:10.749248 23520 raft_consensus.cc:3053] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Stepping down as leader of term 9
I20250831 13:59:10.827855 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5040bfaecba143f4af6334f314734a46" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.828297 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.827875 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9970f4ff4e4b4d8b8beceaf00d20c3f6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:10.827912 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:10.829448 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
I20250831 13:59:10.829210 20642 raft_consensus.cc:1238] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:10.830001 20650 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Advancing to term 12
I20250831 13:59:10.794749 23464 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:10.826925 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:10.833768 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.824961 20616 leader_election.cc:304] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters: 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.836513 20648 raft_consensus.cc:1273] T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20250831 13:59:10.769097 21201 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ab7ade51fa134a1b814fab748f01d43d" is_pre_election: true
I20250831 13:59:10.840817 20498 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:10.861465 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb59414ade0543ef8c7f047104f39576" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.861716 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.867326 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.869680 20574 raft_consensus.cc:2466] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 8.
I20250831 13:59:10.872989 20501 raft_consensus.cc:2366] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9.
I20250831 13:59:10.873131 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e9d8af1af01e4b7f83bc2aaded81ccbf" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.874003 20571 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:10.834998 23513 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250831 13:59:10.827943 23499 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20250831 13:59:10.800495 23315 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20250831 13:59:10.876552 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:10.884128 20496 raft_consensus.cc:2366] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:59:10.877292 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.885313 20498 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:10.836580 23392 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250831 13:59:10.795063 23083 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.889070 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.889763 20493 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 6.
I20250831 13:59:10.880122 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.828115 23520 raft_consensus.cc:738] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:10.802102 23511 consensus_queue.cc:1046] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.070s
W20250831 13:59:10.887313 20543 leader_election.cc:336] T 00c91580fd1b4c45a280862eedb8e1ac P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.858s (SENT)
W20250831 13:59:10.907466 20541 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:10.927184 20643 raft_consensus.cc:2466] T bf7dfe5303d444eeb5f33f78d248eaec P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 7.
I20250831 13:59:10.827538 21174 leader_election.cc:304] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:10.879004 23513 raft_consensus.cc:491] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:10.883013 23499 raft_consensus.cc:491] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:10.937076 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.887035 23392 raft_consensus.cc:491] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250831 13:59:10.944056 20498 raft_consensus.cc:2366] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 11. Current term is 12.
I20250831 13:59:10.942253 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.858800 23464 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.874935 23380 consensus_queue.cc:1035] T c8699373860740a9b912a1ce516e1510 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20250831 13:59:10.924270 23475 raft_consensus.cc:2802] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:10.876180 23466 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20250831 13:59:10.970830 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "099d69058d8b46999f57c3142f80acd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:10.971962 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bca7755948d54e9ab6dbefcbf93c927f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:10.942724 23513 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:10.894096 23402 consensus_queue.cc:1046] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20250831 13:59:10.942790 23499 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:59:10.949800 23392 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:10.950299 22873 raft_consensus.cc:2747] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:10.946508 23511 raft_consensus.cc:3053] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Stepping down as leader of term 6
I20250831 13:59:10.938522 23531 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:10.948779 23467 raft_consensus.cc:491] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:10.991014 20650 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 44d163094add49049cc749547d891602 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 12.
W20250831 13:59:10.950541 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 135962us
W20250831 13:59:10.936461 20447 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 90a0a56228d04660b3cb351485be6593 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20250831 13:59:11.016647 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "795afdca08334d5aa9a9b944be40f34f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.017611 20502 raft_consensus.cc:2366] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:59:10.980170 23475 raft_consensus.cc:695] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:10.999944 23400 raft_consensus.cc:3053] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Stepping down as leader of term 5
I20250831 13:59:11.016201 23467 raft_consensus.cc:513] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:11.020395 20447 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 161195us
I20250831 13:59:11.038512 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.051390 20502 raft_consensus.cc:2408] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20250831 13:59:10.999003 23511 raft_consensus.cc:738] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:11.021628 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c47fb589f98457ab81d63e6b9d19fd0" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
W20250831 13:59:11.036197 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 160080us
I20250831 13:59:11.046248 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00c91580fd1b4c45a280862eedb8e1ac" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.022572 23474 leader_election.cc:290] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:11.040938 23400 raft_consensus.cc:738] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:11.066730 23000 raft_consensus.cc:513] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.069409 23000 leader_election.cc:290] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:11.070403 20645 raft_consensus.cc:2391] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 58aded50ab0d4609a053647eb11b1f48 in current term 8: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:11.071071 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f52ce7625424dcc943a573347b456bb" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.071311 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.072008 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad68e3a13b9e479791179f35e91bea4b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.072656 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.073211 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1aba213b27e4f4aba685d97e9ba1626" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.073596 20496 raft_consensus.cc:2391] T 21eaf12e87504154aa74e46187b415ae P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:11.073997 20499 raft_consensus.cc:2391] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 9: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:11.074667 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.075624 20501 raft_consensus.cc:2366] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:59:11.076676 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2d4d2e85a054797ae54ac949008c1f1" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:11.078011 20543 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:11.079406 20502 raft_consensus.cc:1238] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48 [term 10 LEADER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 9. Current term is 10. Ops: []
W20250831 13:59:11.082316 20543 leader_election.cc:336] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.815s (SENT)
I20250831 13:59:11.083530 23533 raft_consensus.cc:491] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:11.084040 23533 raft_consensus.cc:513] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.086206 23533 leader_election.cc:290] T 8b2d32573f02435681e911c3badba77b P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:11.087800 23467 leader_election.cc:290] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:11.088973 23464 leader_election.cc:290] T a7956988b57844cfad1028024ac745e6 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:11.089682 23083 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:11.094957 20498 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.095662 20498 raft_consensus.cc:2366] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 8. Current term is 9.
I20250831 13:59:11.096683 20500 raft_consensus.cc:1238] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Rejecting Update request from peer 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8. Ops: []
I20250831 13:59:11.097790 23475 consensus_queue.cc:237] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:10.985313 23520 consensus_queue.cc:260] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.099284 23520 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:11.108518 23511 consensus_queue.cc:260] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.109911 23511 raft_consensus.cc:3058] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:11.013468 23531 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.072114 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.115592 20493 raft_consensus.cc:2366] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 6. Current term is 7.
I20250831 13:59:11.048863 23466 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.067s
I20250831 13:59:11.116756 23400 consensus_queue.cc:260] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.125375 20543 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:11.129704 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "79dc07c2112a46fe82d7250aa95bd6dc" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.132658 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24ba1ebe867e4fbb9b5f869fbfd4b489" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.134651 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:11.136140 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:11.137602 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e3ed7784853456bbe218b149399b557" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.138484 20500 raft_consensus.cc:2366] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:59:11.143160 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f29595dd2c044618bd6cb4517fbc5c7b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
W20250831 13:59:11.148250 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:11.129844 23381 raft_consensus.cc:2747] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250831 13:59:11.153710 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6dd87c9794194ce3a9cb6e3b933c3215" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.154336 20575 raft_consensus.cc:3058] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Advancing to term 12
W20250831 13:59:11.158383 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 76131.9us
W20250831 13:59:11.159018 20543 leader_election.cc:336] T 21eaf12e87504154aa74e46187b415ae P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.870s (SENT)
I20250831 13:59:11.159878 20574 raft_consensus.cc:1273] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20250831 13:59:11.162237 20576 raft_consensus.cc:1273] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20250831 13:59:11.164040 20543 leader_election.cc:336] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.909s (SENT)
I20250831 13:59:11.164439 20543 leader_election.cc:304] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:11.166944 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:11.168358 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 343460us
I20250831 13:59:11.172277 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:11.173048 20572 raft_consensus.cc:2466] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 9.
W20250831 13:59:11.180112 20543 leader_election.cc:336] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.996s (SENT)
I20250831 13:59:11.180625 20543 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:11.181998 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:11.176082 20568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.012s sys 0.013s
W20250831 13:59:11.183064 20568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s user 0.012s sys 0.013s
I20250831 13:59:11.173918 23381 raft_consensus.cc:2747] T e9d8af1af01e4b7f83bc2aaded81ccbf P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:11.189694 20575 raft_consensus.cc:2466] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 12.
I20250831 13:59:11.173800 23531 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:11.198014 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:11.181908 23533 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:59:11.217128 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 392861us
W20250831 13:59:11.218315 20543 leader_election.cc:336] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.008s (SENT)
I20250831 13:59:11.232996 20543 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:11.215919 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:11.234069 20467 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:11.230446 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 393260us
I20250831 13:59:11.249890 20466 leader_election.cc:304] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
W20250831 13:59:11.250840 20465 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:11.251781 20615 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:11.253727 20467 leader_election.cc:304] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
W20250831 13:59:11.262859 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 93633.7us
I20250831 13:59:11.266628 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:11.267341 20647 raft_consensus.cc:2466] T 192560b5dd5040279d722ef31a62c940 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 7.
I20250831 13:59:11.268034 20467 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.261902 23264 raft_consensus.cc:2747] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 13:59:11.271483 23513 raft_consensus.cc:513] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:11.272150 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 64813.8us
W20250831 13:59:11.272729 20543 leader_election.cc:336] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.012s (SENT)
I20250831 13:59:11.273094 20543 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:11.273419 23513 leader_election.cc:290] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:11.273860 23190 raft_consensus.cc:2747] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250831 13:59:11.274144 20543 leader_election.cc:336] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.024s (SENT)
I20250831 13:59:11.275303 23499 raft_consensus.cc:513] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.277354 23499 leader_election.cc:290] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:11.279453 23539 log.cc:927] Time spent T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20250831 13:59:11.279274 23392 raft_consensus.cc:513] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.281348 23392 leader_election.cc:290] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
W20250831 13:59:11.256768 23500 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20250831 13:59:11.282398 20645 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b2d32573f02435681e911c3badba77b" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:11.274612 20543 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:11.286914 20541 leader_election.cc:304] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:11.287917 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1700) took 1188 ms. Trace:
I20250831 13:59:11.257169 23468 raft_consensus.cc:2802] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election won for term 8
W20250831 13:59:11.292407 21174 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 252606us
I20250831 13:59:11.258929 23531 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:11.296169 20467 leader_election.cc:304] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.288447 20577 rpcz_store.cc:276] 0831 13:59:10.099965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:10.101667 (+ 1702us) service_pool.cc:225] Handling call
0831 13:59:10.126124 (+ 24457us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:59:10.137162 (+ 11038us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:10.137206 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:59:10.691936 (+554730us) spinlock_profiling.cc:243] Waited 2.52 ms on lock 0x7b2c00000c88. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bb3199a1 00007fb9bb328fd2 00007fb9bc9b2021 00007fb9bc9763fa 00007fb9bc9757fe 00007fb9c18d2c89 00007fb9baf5fb44 00007fb9baf5fad2 00007fb9baf5fa01 00007fb9baf5f97c 00007fb9baf5ec32
0831 13:59:10.692592 (+ 656us) log.cc:844] Serialized 52 byte log entry
0831 13:59:11.043420 (+350828us) raft_consensus.cc:1643] Marking committed up to 2
0831 13:59:11.077928 (+ 34508us) log.cc:844] Serialized 14 byte log entry
0831 13:59:11.078207 (+ 279us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:11.078295 (+ 88us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:11.282141 (+203846us) raft_consensus.cc:1683] finished
0831 13:59:11.287711 (+ 5570us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:11.287858 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":34855,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":8256,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":7334,"spinlock_wait_cycles":5792000,"thread_start_us":349941,"threads_started":1,"wal-append.queue_time_us":575582}
I20250831 13:59:11.295807 20466 leader_election.cc:304] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:11.305064 20467 leader_election.cc:304] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.305639 23468 raft_consensus.cc:695] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:11.306141 23464 raft_consensus.cc:2747] T a05a337dd9e04dec9e8874b1d6c4d114 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250831 13:59:11.306650 20466 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48; no voters:
I20250831 13:59:11.310240 23499 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20250831 13:59:11.312178 21175 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.852s user 0.006s sys 0.019s
I20250831 13:59:11.314363 19366 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250831 13:59:11.341096 21175 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s user 0.007s sys 0.019s
W20250831 13:59:11.341708 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 884045us
I20250831 13:59:11.314848 20467 leader_election.cc:304] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.307053 23467 raft_consensus.cc:2802] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:11.294812 23190 raft_consensus.cc:2802] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250831 13:59:11.294363 23264 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:11.284806 23435 consensus_queue.cc:1046] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20250831 13:59:11.343441 20467 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.297631 23533 raft_consensus.cc:2747] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:59:11.340276 20543 leader_election.cc:336] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.427s (SENT)
I20250831 13:59:11.291453 23505 consensus_queue.cc:1035] T e93846478efb47a1b008fc73d5d25914 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
W20250831 13:59:11.350629 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 64416.8us
I20250831 13:59:11.351795 20569 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "512ab0e0a72d4872b147e21acc0832fd" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.353682 20569 raft_consensus.cc:3058] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:11.354622 20543 leader_election.cc:304] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:11.317317 23380 raft_consensus.cc:2747] T bb59414ade0543ef8c7f047104f39576 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20250831 13:59:11.371534 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 61674.8us
I20250831 13:59:11.301017 23482 consensus_queue.cc:1035] T bca7755948d54e9ab6dbefcbf93c927f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20250831 13:59:11.348884 23377 raft_consensus.cc:2802] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Leader election won for term 12
I20250831 13:59:11.374284 20467 leader_election.cc:304] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.354732 23400 raft_consensus.cc:3058] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Advancing to term 6
I20250831 13:59:11.349848 23551 raft_consensus.cc:2747] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 13:59:11.384071 20467 leader_election.cc:304] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.349918 23190 raft_consensus.cc:491] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:11.387491 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.392412 20496 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:11.386400 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.398820 20467 leader_election.cc:304] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.400606 20502 raft_consensus.cc:1273] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20250831 13:59:11.401088 20569 raft_consensus.cc:2466] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7ade51fa134a1b814fab748f01d43d in term 10.
I20250831 13:59:11.378173 23475 raft_consensus.cc:491] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:11.402168 21175 leader_election.cc:304] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, ab7ade51fa134a1b814fab748f01d43d; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:11.402518 23475 raft_consensus.cc:513] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.406549 23475 leader_election.cc:290] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:11.408385 20577 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20250831 13:59:11.375051 23556 log.cc:927] Time spent T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
W20250831 13:59:11.411860 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 93335.3us
W20250831 13:59:11.412472 20543 leader_election.cc:336] T 501e1c372209405c80034fe2411aaa22 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.473s (SENT)
I20250831 13:59:11.400892 20502 pending_rounds.cc:85] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 4
I20250831 13:59:11.413153 20502 pending_rounds.cc:107] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20250831 13:59:11.413436 20502 raft_consensus.cc:2887] T 8de1861ba5b5442381501e5b214a51f9 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:11.372618 23531 raft_consensus.cc:2747] T 5040bfaecba143f4af6334f314734a46 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250831 13:59:11.417084 20495 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.422593 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.423238 20499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.408478 20500 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:11.428323 20497 raft_consensus.cc:1238] T d1aba213b27e4f4aba685d97e9ba1626 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:11.428676 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c933fb1eb5c54d0b8b0a6cc59ba2aef5" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:11.374768 23381 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:11.445201 20501 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.366067 23474 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20250831 13:59:11.381508 23533 raft_consensus.cc:491] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:11.450363 20501 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:11.397615 23190 raft_consensus.cc:3058] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:11.455964 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.456388 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.456997 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.457273 20502 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Advancing to term 8
I20250831 13:59:11.458257 20493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.457981 20568 raft_consensus.cc:1238] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20250831 13:59:11.460248 20493 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:11.460983 20568 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.461544 20568 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:59:11.462404 20575 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Advancing to term 9
I20250831 13:59:11.374768 23535 consensus_queue.cc:1035] T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.071s
I20250831 13:59:11.469130 20572 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d512286b29841fbaba3df87e23ceff4" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.469885 20572 raft_consensus.cc:3058] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:11.476150 20575 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:11.477259 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bf7dfe5303d444eeb5f33f78d248eaec" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:11.408599 23377 raft_consensus.cc:695] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [term 12 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:11.477627 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.478578 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d9ed9ad66d8047f4af53abfa0192ff5a" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.479319 20570 raft_consensus.cc:3058] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Advancing to term 7
I20250831 13:59:11.482226 20497 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.484227 20577 raft_consensus.cc:2391] T bf7dfe5303d444eeb5f33f78d248eaec P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 7: Already voted for candidate 58aded50ab0d4609a053647eb11b1f48 in this term.
I20250831 13:59:11.485618 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c85d8618b084c51b4e509df8650069d" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:11.487969 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 89241.1us
I20250831 13:59:11.493551 20576 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.445802 23380 raft_consensus.cc:2747] T da27681405bf47e28acb6b95258c0b29 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250831 13:59:11.485049 20494 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3359826124a94225a4cd8932ecb7fcff" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.494290 20576 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:59:11.497998 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca82b3cf6a194875b3a6e31278e37e51" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
I20250831 13:59:11.498783 20574 raft_consensus.cc:3058] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Advancing to term 10
I20250831 13:59:11.499832 20467 leader_election.cc:304] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58aded50ab0d4609a053647eb11b1f48; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:11.377262 23554 consensus_queue.cc:1035] T 62fa3a4f7ea043fdbcffdcc84f9c28c0 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20250831 13:59:11.452364 23551 raft_consensus.cc:2747] T 79dc07c2112a46fe82d7250aa95bd6dc P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 13:59:11.511365 20499 raft_consensus.cc:1273] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20250831 13:59:11.517293 20499 pending_rounds.cc:85] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 4
I20250831 13:59:11.517575 20499 pending_rounds.cc:107] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20250831 13:59:11.517868 20499 raft_consensus.cc:2887] T 501e1c372209405c80034fe2411aaa22 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:11.511669 20497 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20250831 13:59:11.511804 20575 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20bf19e3f9b04d0b9d20ce81aa982894" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786"
W20250831 13:59:11.516149 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 103698us
I20250831 13:59:11.518504 20497 pending_rounds.cc:85] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: Aborting all ops after (but not including) 7
I20250831 13:59:11.518781 20497 pending_rounds.cc:107] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48: Aborting uncommitted NO_OP operation due to leader change: 7.8
W20250831 13:59:11.518850 20543 leader_election.cc:336] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 2.643s (SENT)
I20250831 13:59:11.519042 20497 raft_consensus.cc:2887] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250831 13:59:11.519322 20543 leader_election.cc:304] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:11.520025 20575 raft_consensus.cc:3058] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Advancing to term 8
I20250831 13:59:11.524257 20502 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:11.534377 20574 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:11.529492 20577 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:11.534971 20575 raft_consensus.cc:2466] T 20bf19e3f9b04d0b9d20ce81aa982894 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 8.
I20250831 13:59:11.541929 20568 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:11.535890 20570 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:11.543658 20494 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250831 13:59:11.504681 23466 raft_consensus.cc:2802] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Leader election won for term 10
I20250831 13:59:11.548390 20577 raft_consensus.cc:2391] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44d163094add49049cc749547d891602 in current term 9: Already voted for candidate 212b66976f874aa682cae0fb77928786 in this term.
I20250831 13:59:11.504412 23531 raft_consensus.cc:491] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:11.455189 23381 raft_consensus.cc:513] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.515669 23566 raft_consensus.cc:2747] T f10d72d324374615bbfb06fd2c0e548d P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250831 13:59:11.527104 23536 raft_consensus.cc:491] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:11.567001 21201 raft_consensus.cc:1273] T 346d047c73c648a5aab2f44efa73aee7 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 13:59:11.471697 23533 raft_consensus.cc:513] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:11.570191 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 220948us
I20250831 13:59:11.572549 20574 raft_consensus.cc:1238] T 512ab0e0a72d4872b147e21acc0832fd P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Rejecting Update request from peer 44d163094add49049cc749547d891602 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20250831 13:59:11.543377 23511 raft_consensus.cc:491] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250831 13:59:11.580474 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 61657.9us
I20250831 13:59:11.580660 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b9f875f43094448cbafb11704a281c08" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.614319 20502 raft_consensus.cc:3058] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Advancing to term 7
I20250831 13:59:11.583840 20572 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:11.585170 20576 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 11.
I20250831 13:59:11.545243 23467 raft_consensus.cc:695] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [term 7 LEADER]: Becoming Leader. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
I20250831 13:59:11.503556 23468 consensus_queue.cc:237] T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:11.598318 21175 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 196227us
I20250831 13:59:11.529208 23380 raft_consensus.cc:2747] T ad68e3a13b9e479791179f35e91bea4b P 58aded50ab0d4609a053647eb11b1f48 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250831 13:59:11.625936 23577 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20250831 13:59:11.627542 23467 consensus_queue.cc:237] T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:11.579867 23543 log.cc:927] Time spent T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.285s user 0.000s sys 0.001s
I20250831 13:59:11.633833 20502 raft_consensus.cc:2466] T b9f875f43094448cbafb11704a281c08 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:11.569093 23482 raft_consensus.cc:2747] T 1e3ed7784853456bbe218b149399b557 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:11.607173 20577 raft_consensus.cc:1273] T 62d7feaf5302450caacdf561ce6f06d8 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250831 13:59:11.579382 23487 raft_consensus.cc:491] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:11.653398 20500 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ccee615469564d43800cb6f12f325e22" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.654264 20500 raft_consensus.cc:3058] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Advancing to term 11
I20250831 13:59:11.622764 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "501e1c372209405c80034fe2411aaa22" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:11.624531 20643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a7956988b57844cfad1028024ac745e6" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:11.632298 20647 raft_consensus.cc:1273] T 2c47fb589f98457ab81d63e6b9d19fd0 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Refusing update from remote peer 58aded50ab0d4609a053647eb11b1f48: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20250831 13:59:11.603044 23466 raft_consensus.cc:695] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [term 10 LEADER]: Becoming Leader. State: Replica: ab7ade51fa134a1b814fab748f01d43d, State: Running, Role: LEADER
I20250831 13:59:11.604987 23550 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:11.660607 20644 raft_consensus.cc:1238] T 995d9ef0c84a4aae8581ab92c0607fdf P 44d163094add49049cc749547d891602 [term 9 LEADER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
I20250831 13:59:11.661536 20646 raft_consensus.cc:1238] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Rejecting Update request from peer 58aded50ab0d4609a053647eb11b1f48 for earlier term 8. Current term is 9. Ops: []
W20250831 13:59:11.671205 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:11.641256 23552 consensus_queue.cc:1035] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20250831 13:59:11.596151 23551 consensus_queue.cc:1035] T 8b2d32573f02435681e911c3badba77b P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20250831 13:59:11.680267 20502 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e8142901f0248e7b0565a5be0dbda2b" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48"
I20250831 13:59:11.680877 20502 raft_consensus.cc:3058] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
W20250831 13:59:11.674358 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 110573us
I20250831 13:59:11.600657 23531 raft_consensus.cc:513] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.604553 23536 raft_consensus.cc:513] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
W20250831 13:59:11.679080 23553 log.cc:927] Time spent T 099d69058d8b46999f57c3142f80acd0 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.290s user 0.001s sys 0.000s
I20250831 13:59:11.598541 23377 consensus_queue.cc:237] T 6dd87c9794194ce3a9cb6e3b933c3215 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.691648 20494 raft_consensus.cc:1273] T 90a0a56228d04660b3cb351485be6593 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Refusing update from remote peer ab7ade51fa134a1b814fab748f01d43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20250831 13:59:11.630018 23511 raft_consensus.cc:513] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.690727 20616 leader_election.cc:304] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:59:11.682725 23545 log.cc:927] Time spent T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.306s user 0.000s sys 0.001s
W20250831 13:59:11.696465 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:11.697206 23083 raft_consensus.cc:2802] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:11.697746 23083 raft_consensus.cc:695] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [term 9 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:11.698622 23083 consensus_queue.cc:237] T 0e8142901f0248e7b0565a5be0dbda2b P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.701767 23474 consensus_queue.cc:1035] T 346d047c73c648a5aab2f44efa73aee7 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20250831 13:59:11.705284 20616 leader_election.cc:304] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:59:11.688898 23558 log.cc:927] Time spent T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.280s user 0.001s sys 0.000s
I20250831 13:59:11.708956 20616 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:11.689891 23482 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:11.674000 23487 raft_consensus.cc:513] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.676837 23550 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.723345 20543 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, ab7ade51fa134a1b814fab748f01d43d
I20250831 13:59:11.711138 23474 raft_consensus.cc:2802] T ca82b3cf6a194875b3a6e31278e37e51 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader election won for term 10
I20250831 13:59:11.716455 23564 raft_consensus.cc:2802] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 8 FOLLOWER]: Leader election won for term 8
I20250831 13:59:11.775107 20616 leader_election.cc:304] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:11.778194 20571 raft_consensus.cc:1273] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20250831 13:59:11.844797 20617 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:11.780779 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.132:35173, user_credentials={real_user=slave}} blocked reactor thread for 57542.6us
I20250831 13:59:11.813882 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:11.830052 20496 raft_consensus.cc:2466] T 6d512286b29841fbaba3df87e23ceff4 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
I20250831 13:59:11.830380 20493 raft_consensus.cc:2466] T ca82b3cf6a194875b3a6e31278e37e51 P 58aded50ab0d4609a053647eb11b1f48 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 10.
I20250831 13:59:11.846640 20501 raft_consensus.cc:2466] T d9ed9ad66d8047f4af53abfa0192ff5a P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 7.
I20250831 13:59:11.860085 20502 raft_consensus.cc:2466] T 0e8142901f0248e7b0565a5be0dbda2b P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
W20250831 13:59:11.849838 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 341801us
W20250831 13:59:11.853842 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 78818.9us
I20250831 13:59:11.861346 20467 leader_election.cc:304] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:11.853745 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca1477952e8c40adb14c71db5a7a4b92" candidate_uuid: "ab7ade51fa134a1b814fab748f01d43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:11.889804 20496 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9bc5bbba290e48548aa3237b4d8e1bd1" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "58aded50ab0d4609a053647eb11b1f48" is_pre_election: true
I20250831 13:59:11.898234 20496 raft_consensus.cc:2466] T 9bc5bbba290e48548aa3237b4d8e1bd1 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 9.
W20250831 13:59:11.790830 23520 consensus_meta.cc:287] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 0.691s user 0.010s sys 0.001s
W20250831 13:59:11.883689 20543 leader_election.cc:336] T 2c47fb589f98457ab81d63e6b9d19fd0 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.598s (SENT)
W20250831 13:59:11.908146 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 63456.3us
I20250831 13:59:11.909480 20617 leader_election.cc:304] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:11.902766 20573 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.000s sys 0.023s
W20250831 13:59:11.910794 20573 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.000s sys 0.023s
W20250831 13:59:11.807081 23565 log.cc:927] Time spent T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
I20250831 13:59:11.843725 23505 raft_consensus.cc:491] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:11.805209 23536 leader_election.cc:290] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:11.853792 23381 leader_election.cc:290] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
W20250831 13:59:11.986063 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 128055us
I20250831 13:59:11.871989 23533 leader_election.cc:290] T 7562a669789845418660f6f1c3109b1e P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:11.799484 23570 raft_consensus.cc:491] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:11.897567 23499 raft_consensus.cc:2802] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:11.803397 23588 raft_consensus.cc:2747] T 6dd87c9794194ce3a9cb6e3b933c3215 P 212b66976f874aa682cae0fb77928786 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250831 13:59:11.772795 23482 raft_consensus.cc:513] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:11.791433 23380 consensus_queue.cc:1046] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.188s
W20250831 13:59:11.998399 20543 leader_election.cc:336] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131): Timed out: RequestConsensusVote RPC to 127.18.233.129:35131 timed out after 1.601s (SENT)
I20250831 13:59:11.776208 23435 consensus_queue.cc:1046] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.207s
W20250831 13:59:11.999874 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 138305us
I20250831 13:59:11.794941 23568 consensus_queue.cc:1035] T 90a0a56228d04660b3cb351485be6593 P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.217s
I20250831 13:59:12.010429 20543 leader_election.cc:304] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:11.979804 23402 raft_consensus.cc:2747] T 1c85d8618b084c51b4e509df8650069d P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20250831 13:59:12.007369 20616 leader_election.cc:304] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:12.008118 23381 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250831 13:59:12.009637 23549 raft_consensus.cc:991] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602: : Instructing follower 58aded50ab0d4609a053647eb11b1f48 to start an election
I20250831 13:59:11.773051 23566 consensus_queue.cc:1035] T 2c47fb589f98457ab81d63e6b9d19fd0 P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.211s
I20250831 13:59:12.024943 23264 raft_consensus.cc:3053] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:12.007308 23392 raft_consensus.cc:2747] T c933fb1eb5c54d0b8b0a6cc59ba2aef5 P 44d163094add49049cc749547d891602 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250831 13:59:12.011554 20466 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 236348us
I20250831 13:59:11.787123 23466 consensus_queue.cc:237] T 512ab0e0a72d4872b147e21acc0832fd P ab7ade51fa134a1b814fab748f01d43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:12.018419 23499 raft_consensus.cc:695] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:59:12.024796 23571 log.cc:927] Time spent T 79dc07c2112a46fe82d7250aa95bd6dc P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
I20250831 13:59:12.011310 23531 leader_election.cc:290] T 223426ea4b53470699bceb4fb06dd992 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:12.035518 20616 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:12.072166 20616 leader_election.cc:304] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:59:12.028198 20541 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.131:45067, user_credentials={real_user=slave}} blocked reactor thread for 219631us
I20250831 13:59:12.027022 23511 leader_election.cc:290] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:12.076776 21175 leader_election.cc:304] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab7ade51fa134a1b814fab748f01d43d; no voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602
I20250831 13:59:12.028920 23264 raft_consensus.cc:738] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:59:12.080471 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 219192us
I20250831 13:59:12.016147 23505 raft_consensus.cc:513] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:12.082329 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:12.053566 23478 raft_consensus.cc:1079] T f10d72d324374615bbfb06fd2c0e548d P 44d163094add49049cc749547d891602 [term 4 LEADER]: Signalling peer 58aded50ab0d4609a053647eb11b1f48 to start an election
I20250831 13:59:12.004606 23570 raft_consensus.cc:513] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.087318 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "223426ea4b53470699bceb4fb06dd992" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:12.087654 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 100963us
I20250831 13:59:12.089118 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "da27681405bf47e28acb6b95258c0b29" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:12.089869 20646 raft_consensus.cc:2466] T da27681405bf47e28acb6b95258c0b29 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 212b66976f874aa682cae0fb77928786 in term 10.
I20250831 13:59:12.091349 20541 leader_election.cc:304] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
W20250831 13:59:12.092307 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 182899us
I20250831 13:59:12.088730 20648 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7562a669789845418660f6f1c3109b1e" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:12.071507 23588 raft_consensus.cc:491] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
W20250831 13:59:12.084522 23190 consensus_meta.cc:287] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786: Time spent flushing consensus metadata: real 0.590s user 0.011s sys 0.001s
I20250831 13:59:12.086861 20467 leader_election.cc:304] T 20bf19e3f9b04d0b9d20ce81aa982894 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48; no voters: 44d163094add49049cc749547d891602
I20250831 13:59:12.082751 23000 raft_consensus.cc:2802] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 11 FOLLOWER]: Leader election won for term 11
I20250831 13:59:12.077033 23550 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:12.081262 23083 raft_consensus.cc:2802] T 6d512286b29841fbaba3df87e23ceff4 P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election won for term 9
I20250831 13:59:12.085044 23536 raft_consensus.cc:2747] T ca1477952e8c40adb14c71db5a7a4b92 P ab7ade51fa134a1b814fab748f01d43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20250831 13:59:12.154996 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 58870.6us
I20250831 13:59:12.034971 23582 consensus_queue.cc:1046] T 512ab0e0a72d4872b147e21acc0832fd P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.089s
W20250831 13:59:12.179245 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 60736.3us
I20250831 13:59:12.181631 20617 leader_election.cc:304] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:12.199174 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:12.214896 20500 consensus_meta.cc:287] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48: Time spent flushing consensus metadata: real 0.557s user 0.011s sys 0.000s
I20250831 13:59:12.215366 20500 raft_consensus.cc:2466] T ccee615469564d43800cb6f12f325e22 P 58aded50ab0d4609a053647eb11b1f48 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 11.
I20250831 13:59:12.123660 23583 raft_consensus.cc:2747] T bca7755948d54e9ab6dbefcbf93c927f P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:12.211804 20573 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41400 (request call id 1623) took 1071 ms. Trace:
I20250831 13:59:12.244153 23482 leader_election.cc:290] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:12.189836 23533 raft_consensus.cc:2802] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250831 13:59:12.125591 23190 raft_consensus.cc:513] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.188473 23564 raft_consensus.cc:695] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [term 8 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:12.228477 20573 rpcz_store.cc:276] 0831 13:59:11.140800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:11.140932 (+ 132us) service_pool.cc:225] Handling call
0831 13:59:11.157391 (+ 16459us) raft_consensus.cc:1469] Updating replica for 2 ops
0831 13:59:11.157755 (+ 364us) raft_consensus.cc:1532] Early marking committed up to index 3
0831 13:59:11.295339 (+137584us) log.cc:844] Serialized 14 byte log entry
0831 13:59:11.911172 (+615833us) raft_consensus.cc:1537] Triggering prepare for 2 ops
0831 13:59:11.911855 (+ 683us) log.cc:844] Serialized 52 byte log entry
0831 13:59:11.912029 (+ 174us) raft_consensus.cc:1643] Marking committed up to 5
0831 13:59:12.072792 (+160763us) log.cc:844] Serialized 14 byte log entry
0831 13:59:12.124678 (+ 51886us) log.cc:844] Serialized 14 byte log entry
0831 13:59:12.211346 (+ 86668us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:12.211453 (+ 107us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:12.211513 (+ 60us) raft_consensus.cc:1683] finished
0831 13:59:12.211536 (+ 23us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:12.211727 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":143776,"prepare.queue_time_us":105928,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":8129,"raft.queue_time_us":230705,"raft.run_cpu_time_us":4908,"raft.run_wall_time_us":74048,"spinlock_wait_cycles":53731712,"thread_start_us":873971,"threads_started":4,"wal-append.queue_time_us":229336}
W20250831 13:59:12.271457 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 181945us
I20250831 13:59:12.200799 23585 raft_consensus.cc:491] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:12.219141 23435 consensus_queue.cc:1230] T 995d9ef0c84a4aae8581ab92c0607fdf P 58aded50ab0d4609a053647eb11b1f48 [LEADER]: Peer 44d163094add49049cc749547d891602 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20250831 13:59:12.274724 23380 raft_consensus.cc:491] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:12.188958 23402 consensus_queue.cc:1035] T 8de1861ba5b5442381501e5b214a51f9 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20250831 13:59:12.312403 20543 leader_election.cc:400] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
I20250831 13:59:12.313251 20543 leader_election.cc:403] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250831 13:59:12.265702 23569 raft_consensus.cc:2802] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:12.255770 23533 raft_consensus.cc:491] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:12.217479 23264 consensus_queue.cc:260] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.234061 23392 consensus_queue.cc:1046] T d1aba213b27e4f4aba685d97e9ba1626 P 44d163094add49049cc749547d891602 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
I20250831 13:59:12.316627 23576 raft_consensus.cc:491] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:12.305095 23590 raft_consensus.cc:491] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:12.122785 23588 raft_consensus.cc:513] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 212b66976f874aa682cae0fb77928786 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.331903 23569 raft_consensus.cc:695] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [term 7 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
W20250831 13:59:12.334657 23575 log.cc:927] Time spent T 0c14599239fd4c99b3331692a513a092 P ab7ade51fa134a1b814fab748f01d43d: Append to log took a long time: real 0.320s user 0.000s sys 0.001s
I20250831 13:59:12.320498 23533 raft_consensus.cc:3058] T da27681405bf47e28acb6b95258c0b29 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Advancing to term 10
I20250831 13:59:12.338964 23576 raft_consensus.cc:513] T a7956988b57844cfad1028024ac745e6 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.310024 23585 raft_consensus.cc:513] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:12.412705 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 100436us
I20250831 13:59:12.781998 20543 leader_election.cc:304] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:12.312556 23380 raft_consensus.cc:513] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:12.332312 23578 raft_consensus.cc:2802] T 50faaafc366a4da79cca52e16429da72 P 58aded50ab0d4609a053647eb11b1f48 [term 7 FOLLOWER]: Leader election won for term 7
I20250831 13:59:12.343732 23564 consensus_queue.cc:237] T 20bf19e3f9b04d0b9d20ce81aa982894 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:12.710824 20466 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:12.710985 23591 log.cc:927] Time spent T bca7755948d54e9ab6dbefcbf93c927f P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.521s user 0.002s sys 0.004s
I20250831 13:59:12.711616 23505 leader_election.cc:290] T 21eaf12e87504154aa74e46187b415ae P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:12.712116 23511 raft_consensus.cc:491] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:12.713202 23400 consensus_meta.cc:287] T 9970f4ff4e4b4d8b8beceaf00d20c3f6 P 44d163094add49049cc749547d891602: Time spent flushing consensus metadata: real 1.303s user 0.003s sys 0.027s
I20250831 13:59:12.332696 23579 consensus_queue.cc:1035] T 62d7feaf5302450caacdf561ce6f06d8 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.248s
W20250831 13:59:12.715047 20542 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250831 13:59:12.620862 20569 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.774s user 0.017s sys 0.003s
I20250831 13:59:12.716151 20502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52988 (request call id 1783) took 1003 ms. Trace:
W20250831 13:59:12.818533 20569 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.972s user 0.019s sys 0.004s
I20250831 13:59:12.719198 23190 leader_election.cc:290] T 192560b5dd5040279d722ef31a62c940 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 election: Requested vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:12.335018 23598 raft_consensus.cc:491] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ab7ade51fa134a1b814fab748f01d43d)
I20250831 13:59:12.828166 23598 raft_consensus.cc:513] T 0c14599239fd4c99b3331692a513a092 P 58aded50ab0d4609a053647eb11b1f48 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:12.723667 23264 raft_consensus.cc:3058] T 3359826124a94225a4cd8932ecb7fcff P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Advancing to term 9
I20250831 13:59:12.831336 23594 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:12.831013 23569 consensus_queue.cc:237] T b9f875f43094448cbafb11704a281c08 P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.831804 23594 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.833750 23594 leader_election.cc:290] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:12.821988 23511 raft_consensus.cc:513] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:12.727145 20614 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250831 13:59:12.332800 23608 raft_consensus.cc:2747] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250831 13:59:12.728055 20573 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "50faaafc366a4da79cca52e16429da72"
dest_uuid: "212b66976f874aa682cae0fb77928786"
mode: GRACEFUL
new_leader_uuid: "58aded50ab0d4609a053647eb11b1f48"
from {username='slave'} at 127.0.0.1:49676
I20250831 13:59:12.331466 23603 raft_consensus.cc:3053] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Stepping down as leader of term 8
I20250831 13:59:12.836217 23511 leader_election.cc:290] T 24ba1ebe867e4fbb9b5f869fbfd4b489 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:12.836429 23603 raft_consensus.cc:738] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 58aded50ab0d4609a053647eb11b1f48, State: Running, Role: LEADER
W20250831 13:59:12.737185 23610 log.cc:927] Time spent T bf7dfe5303d444eeb5f33f78d248eaec P 58aded50ab0d4609a053647eb11b1f48: Append to log took a long time: real 0.424s user 0.006s sys 0.001s
I20250831 13:59:12.738328 23402 raft_consensus.cc:491] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250831 13:59:12.740042 23487 leader_election.cc:290] T c8699373860740a9b912a1ce516e1510 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629)
I20250831 13:59:12.743211 20495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52988 (request call id 1759) took 1359 ms. Trace:
I20250831 13:59:12.331418 23612 raft_consensus.cc:491] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 212b66976f874aa682cae0fb77928786 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:12.202329 23499 consensus_queue.cc:237] T d9ed9ad66d8047f4af53abfa0192ff5a P 44d163094add49049cc749547d891602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:12.768855 20577 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.011s sys 0.019s
W20250831 13:59:12.861107 20577 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s user 0.012s sys 0.020s
W20250831 13:59:12.771235 20572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.006s sys 0.004s
W20250831 13:59:12.862702 20572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.746s user 0.007s sys 0.005s
I20250831 13:59:12.862807 20650 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "50faaafc366a4da79cca52e16429da72" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
W20250831 13:59:12.772360 20468 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s user 0.000s sys 0.010s
W20250831 13:59:12.863415 20468 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.000s sys 0.011s
W20250831 13:59:12.771028 20616 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s user 0.006s sys 0.005s
W20250831 13:59:12.865139 20616 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s user 0.006s sys 0.005s
W20250831 13:59:12.865429 20616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 686281us
W20250831 13:59:12.772946 23468 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.009s sys 0.022s
I20250831 13:59:12.814638 23606 raft_consensus.cc:2747] T 6d512286b29841fbaba3df87e23ceff4 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250831 13:59:12.726691 23583 raft_consensus.cc:2747] T 795afdca08334d5aa9a9b944be40f34f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131) with higher term. Message: Invalid argument: T 795afdca08334d5aa9a9b944be40f34f P 58aded50ab0d4609a053647eb11b1f48 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 for earlier term 7. Current term is 8.
W20250831 13:59:12.780272 20499 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s user 0.006s sys 0.025s
I20250831 13:59:12.787593 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "192560b5dd5040279d722ef31a62c940" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602"
W20250831 13:59:12.877447 20499 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s user 0.006s sys 0.025s
W20250831 13:59:12.797853 20617 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.006s sys 0.009s
W20250831 13:59:12.885857 20617 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s user 0.006s sys 0.010s
W20250831 13:59:12.886643 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 705052us
I20250831 13:59:12.887281 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5ce6782647ef46f095f726e552f5ad23" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:12.887634 20617 leader_election.cc:304] T 3359826124a94225a4cd8932ecb7fcff P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44d163094add49049cc749547d891602; no voters: 212b66976f874aa682cae0fb77928786, 58aded50ab0d4609a053647eb11b1f48
W20250831 13:59:12.798149 20494 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.002s sys 0.018s
W20250831 13:59:12.773677 23582 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.003s sys 0.008s
I20250831 13:59:12.799435 20543 leader_election.cc:304] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:12.800002 23531 raft_consensus.cc:491] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:12.905844 23531 raft_consensus.cc:513] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.908447 23531 leader_election.cc:290] T 319db728a12e43e5808e70ff91e5155f P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:12.801803 20647 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a05a337dd9e04dec9e8874b1d6c4d114" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:12.800155 23585 leader_election.cc:290] T a05a337dd9e04dec9e8874b1d6c4d114 P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 58aded50ab0d4609a053647eb11b1f48 (127.18.233.129:35131), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:12.909598 20644 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "319db728a12e43e5808e70ff91e5155f" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:12.910553 20644 raft_consensus.cc:2391] T 319db728a12e43e5808e70ff91e5155f P 44d163094add49049cc749547d891602 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 212b66976f874aa682cae0fb77928786 in current term 9: Already voted for candidate 44d163094add49049cc749547d891602 in this term.
I20250831 13:59:12.839619 23570 leader_election.cc:290] T c719f2a4c3ea4418937337e8e3dd2164 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 212b66976f874aa682cae0fb77928786 (127.18.233.130:44629), 44d163094add49049cc749547d891602 (127.18.233.131:45067)
I20250831 13:59:12.725385 20646 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "58aded50ab0d4609a053647eb11b1f48" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:12.787199 21201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47742 (request call id 131) took 1066 ms. Trace:
I20250831 13:59:12.836233 20573 raft_consensus.cc:604] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Received request to transfer leadership to TS 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:12.918293 20646 raft_consensus.cc:2466] T 1dd4b973f11240dc9b8c480b1a77ccb0 P 44d163094add49049cc749547d891602 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58aded50ab0d4609a053647eb11b1f48 in term 12.
I20250831 13:59:12.819017 20502 rpcz_store.cc:276] 0831 13:59:11.712381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:11.871413 (+159032us) service_pool.cc:225] Handling call
0831 13:59:11.871748 (+ 335us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:12.032785 (+161037us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:12.032829 (+ 44us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:12.038377 (+ 5548us) log.cc:844] Serialized 26 byte log entry
0831 13:59:12.197976 (+159599us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:12.198010 (+ 34us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:12.198073 (+ 63us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:12.715924 (+517851us) raft_consensus.cc:1683] finished
0831 13:59:12.715948 (+ 24us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:12.716091 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":345464,"spinlock_wait_cycles":36133120,"thread_start_us":276043,"threads_started":2,"wal-append.queue_time_us":255972}
W20250831 13:59:12.804191 23083 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.008s sys 0.008s
I20250831 13:59:12.857707 20495 rpcz_store.cc:276] 0831 13:59:11.384049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:11.386977 (+ 2928us) service_pool.cc:225] Handling call
0831 13:59:12.742968 (+1355991us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b54001621d4. stack: 00007fb9b99e2823 00007fb9b99e2720 00007fb9b933c574 00007fb9c2e3e784 00007fb9c2e3e719 00007fb9bc8fa6e1 00007fb9bc97a2aa 00007fb9c18d35c7 00007fb9baf620c4 00007fb9baf62052 00007fb9baf61f81 00007fb9baf61efc 00007fb9baf611b2 00007fb9bad75ccc 00007fb9bad75106 00007fb9bad74aaf
0831 13:59:12.743109 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1570222336}
W20250831 13:59:12.769906 23000 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.022s sys 0.009s
I20250831 13:59:12.810931 23536 raft_consensus.cc:491] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
W20250831 13:59:12.772058 20467 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.004s sys 0.007s
W20250831 13:59:12.939726 20467 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.816s user 0.004s sys 0.007s
W20250831 13:59:12.940346 20467 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.130:44629, user_credentials={real_user=slave}} blocked reactor thread for 853531us
W20250831 13:59:12.774050 20576 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.003s sys 0.034s
I20250831 13:59:12.888187 20574 raft_consensus.cc:2466] T 5ce6782647ef46f095f726e552f5ad23 P 212b66976f874aa682cae0fb77928786 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 4.
W20250831 13:59:12.888617 20494 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.799s user 0.002s sys 0.018s
W20250831 13:59:12.867298 23468 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s user 0.009s sys 0.023s
I20250831 13:59:12.915743 20649 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57698 (request call id 1618) took 1279 ms. Trace:
I20250831 13:59:12.941612 20649 rpcz_store.cc:276] 0831 13:59:11.636429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:11.636561 (+ 132us) service_pool.cc:225] Handling call
0831 13:59:11.636891 (+ 330us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:11.637322 (+ 431us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:11.637382 (+ 60us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:11.637967 (+ 585us) log.cc:844] Serialized 26 byte log entry
0831 13:59:11.850062 (+212095us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:11.850120 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:11.850220 (+ 100us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:12.915488 (+1065268us) raft_consensus.cc:1683] finished
0831 13:59:12.915521 (+ 33us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:12.915673 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1735,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":211659,"threads_started":1,"wal-append.queue_time_us":449077}
I20250831 13:59:12.767876 23590 raft_consensus.cc:513] T e7166435c35c41f580e6cfbdaa2d0c39 P 212b66976f874aa682cae0fb77928786 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
W20250831 13:59:12.794358 23600 log.cc:927] Time spent T 90a0a56228d04660b3cb351485be6593 P 44d163094add49049cc749547d891602: Append to log took a long time: real 0.593s user 0.004s sys 0.032s
W20250831 13:59:12.913367 23549 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.009s sys 0.011s
W20250831 13:59:12.941125 20576 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.044s user 0.003s sys 0.034s
I20250831 13:59:12.918493 20573 raft_consensus.cc:612] T 50faaafc366a4da79cca52e16429da72 P 212b66976f874aa682cae0fb77928786 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250831 13:59:12.843302 23402 raft_consensus.cc:513] T 50faaafc366a4da79cca52e16429da72 P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } }
I20250831 13:59:12.951347 23380 leader_election.cc:290] T 346d047c73c648a5aab2f44efa73aee7 P 58aded50ab0d4609a053647eb11b1f48 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 44d163094add49049cc749547d891602 (127.18.233.131:45067), ab7ade51fa134a1b814fab748f01d43d (127.18.233.132:35173)
W20250831 13:59:12.939404 23000 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s user 0.023s sys 0.009s
I20250831 13:59:12.955236 20576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41404 (request call id 1741) took 1396 ms. Trace:
W20250831 13:59:12.942996 23083 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.009s sys 0.010s
W20250831 13:59:12.952749 20617 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 65232.4us
I20250831 13:59:12.940824 20570 raft_consensus.cc:1273] T 5806109eefc8457d8e9110b9ec030873 P 212b66976f874aa682cae0fb77928786 [term 10 FOLLOWER]: Refusing update from remote peer 44d163094add49049cc749547d891602: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20250831 13:59:12.904400 23582 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.671s user 0.003s sys 0.009s
I20250831 13:59:12.952034 23466 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P ab7ade51fa134a1b814fab748f01d43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:12.946003 20574 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8699373860740a9b912a1ce516e1510" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:12.960103 20576 rpcz_store.cc:276] 0831 13:59:11.559273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0831 13:59:11.629899 (+ 70626us) service_pool.cc:225] Handling call
0831 13:59:11.630217 (+ 318us) raft_consensus.cc:1469] Updating replica for 1 ops
0831 13:59:11.850439 (+220222us) raft_consensus.cc:1532] Early marking committed up to index 0
0831 13:59:11.850481 (+ 42us) raft_consensus.cc:1537] Triggering prepare for 1 ops
0831 13:59:11.896632 (+ 46151us) log.cc:844] Serialized 26 byte log entry
0831 13:59:12.954920 (+1058288us) raft_consensus.cc:1643] Marking committed up to 4
0831 13:59:12.954959 (+ 39us) raft_consensus.cc:1695] Filling consensus response to leader.
0831 13:59:12.955020 (+ 61us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0831 13:59:12.955056 (+ 36us) raft_consensus.cc:1683] finished
0831 13:59:12.955069 (+ 13us) raft_consensus.cc:1689] UpdateReplicas() finished
0831 13:59:12.955188 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42494,"raft.queue_time_us":370765,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":14642,"thread_start_us":1277602,"threads_started":2,"wal-append.queue_time_us":967612}
I20250831 13:59:12.971462 20574 raft_consensus.cc:2466] T c8699373860740a9b912a1ce516e1510 P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44d163094add49049cc749547d891602 in term 6.
W20250831 13:59:12.963168 20543 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.18.233.129:35131, user_credentials={real_user=slave}} blocked reactor thread for 163847us
I20250831 13:59:12.972609 20543 leader_election.cc:304] T 1c85d8618b084c51b4e509df8650069d P 212b66976f874aa682cae0fb77928786 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 212b66976f874aa682cae0fb77928786; no voters: 44d163094add49049cc749547d891602, 58aded50ab0d4609a053647eb11b1f48
I20250831 13:59:12.964740 20570 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1dd4b973f11240dc9b8c480b1a77ccb0" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
I20250831 13:59:12.954735 23644 consensus_queue.cc:1035] T e7166435c35c41f580e6cfbdaa2d0c39 P 44d163094add49049cc749547d891602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20250831 13:59:12.974783 20573 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21eaf12e87504154aa74e46187b415ae" candidate_uuid: "44d163094add49049cc749547d891602" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "212b66976f874aa682cae0fb77928786" is_pre_election: true
W20250831 13:59:12.960013 23474 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.949s user 0.027s sys 0.031s
I20250831 13:59:12.966334 23381 raft_consensus.cc:2747] T 7f52ce7625424dcc943a573347b456bb P 212b66976f874aa682cae0fb77928786 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250831 13:59:12.961324 23000 raft_consensus.cc:695] T ccee615469564d43800cb6f12f325e22 P 44d163094add49049cc749547d891602 [term 11 LEADER]: Becoming Leader. State: Replica: 44d163094add49049cc749547d891602, State: Running, Role: LEADER
I20250831 13:59:12.967129 23583 raft_consensus.cc:491] T 712d44ed33ec4e37b2bef30d18b3495b P 212b66976f874aa682cae0fb77928786 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 44d163094add49049cc749547d891602)
I20250831 13:59:12.968070 23641 raft_consensus.cc:491] T 00c91580fd1b4c45a280862eedb8e1ac P 44d163094add49049cc749547d891602 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 58aded50ab0d4609a053647eb11b1f48)
I20250831 13:59:12.952067 23536 raft_consensus.cc:513] T 2c47fb589f98457ab81d63e6b9d19fd0 P ab7ade51fa134a1b814fab748f01d43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } attrs { replace: true } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131" port: 45067 } } peers { permanent_uuid: "ab7ade51fa134a1b814fab748f01d43d" member_type: VOTER last_known_addr { host: "127.18.233.132" port: 35173 } attrs { promote: false } }
I20250831 13:59:12.969302 20649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7166435c35c41f580e6cfbdaa2d0c39" candidate_uuid: "212b66976f874aa682cae0fb77928786" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "44d163094add49049cc749547d891602" is_pre_election: true
I20250831 13:59:12.969475 20616 leader_election.cc:304] T 5ce6782647ef46f095f726e552f5ad23 P 44d163094add49049cc749547d891602 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212b66976f874aa682cae0fb77928786, 44d163094add49049cc749547d891602; no voters:
I20250831 13:59:12.970721 20577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34436 (request call id 97) took 1193 ms. Trace:
I20250831 13:59:12.951025 23603 consensus_queue.cc:260] T 319db728a12e43e5808e70ff91e5155f P 58aded50ab0d4609a053647eb11b1f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58aded50ab0d4609a053647eb11b1f48" member_type: VOTER last_known_addr { host: "127.18.233.129" port: 35131 } } peers { permanent_uuid: "212b66976f874aa682cae0fb77928786" member_type: VOTER last_known_addr { host: "127.18.233.130" port: 44629 } } peers { permanent_uuid: "44d163094add49049cc749547d891602" member_type: VOTER last_known_addr { host: "127.18.233.131"